Embedded network programming

Unfortunately, dear readers, it would appear that a long drafted networking post I wrote over the last week has vanished from this site while it was still in the draft stage.  Thus I'll endeavor to rebuild it to get the salient points as I'm very excited about what's coming up and want to get through the prerequisites which include network programming and requirements for talking between embedded devices.
So let's dive right in and start with a quick chat about network programming in general.  Let's start by thinking about some context.  You've got these two devices and you want them to talk to each other.  Well how would two humans figure out how to talk to each other.  The steps are actually modeled after human communication and I'd posit that all communication happens the same way.
First step, identify that there's someone else to talk to.  Let's say you're in a room full of people, in order to not be silent, you must be able to recognize that someone else exists, or you'll only ever talk to yourself.
Having spotted the other communicator, you must find an agreed upon communication protocol, i.e. if one of you is expecting to shake hands and the other is expecting to say hello first, there's a slight protocol negotiation that must be sorted to advance into conversation mode.  However, if one protocol isn't understood such as one speaks in a foreign language that the other doesn't understand, then communication upon that protocol breaks down and the only way to overcome this communications breakdown is to establish a new protocol both sides can understand.
Once the communication protocol is understood, one can pass information back and forth still following the social norms of the communication standard protocol.
At the end of the communication, at least one side of the communication must leave and end the communication.
Odd inspection of a common human activity aside, this is the format that all communications fit within.  So let's apply this to the intended target of network programming.  Assume there are two devices one would like to have communicate.  First step is to have the devices discover each other.  So we're going to blast radio waves out and say "I'm device 1 anyone out there?"  The interesting thing here is that the device identifies itself by some name "1".  This is important to us because it explains how would we know where a communication came from and where it needs to go back to.
Now imagine the communication is happening in a crowded room and you want to have just one pair talking to each other.  You'd have to ensure that the listener knows you're talking to them.  But radio waves equally go out to everyone.  So by necessity, all communication must have an understood name of recipient and name of transmitter prepended to it.  If Alice were talking to Bob, Alice would have to say Bob, this is Alice, hello.  Bob would reply with, Alice, this is Bob, I heard you, hi yourself; so on and so forth.  That is essentially the way that all network communications work "on the wire" or over the air.
So now that we have this naming convention of device "1" is named "1" and second device is "2" it follows that the next n devices will get a new name of n+1.  There's billions of devices in the world and if we want to connect all of them and have all of them able to talk to each other, this naming convention would be a mess.  It especially would be a mess if you are wanting to just have two devices talk to each other in a small room.  You still would have to say the name of the device and your name to have a chat so giving some 16 digit unique number just to say hi is not exactly optimal when for that given small room, renaming the devices dynamically "1" and "2" would be simpler.
This is where the concept of local networking came into play.  The idea of that is in a local network, names can be simpler.
The way this local networking thing works is that there's a router that keeps knowledge of the numbered internal names through a Named Address Table and the larger internet as a whole.  The router will take communication from the internal name and use the network's single name to talk to all other networks.  So this is the funnel and makes it so the local network communication can be manageable.
Now as part of this, local network to larger network thing, there's a standard to the naming convention that allows the Named Address Table to maintain the same sized names, or Internet Protocol addresses.  This is an optimization in the lookup time for the routers and results in faster communication in both the smaller Local Area Network and the larger Wide Area Network.  This standard is called IPv4 it has 4 sets of three numbers ranging from 0 to 255.  The local class C network, by convention is set to 192.168.x.y where x is usually 0, 1, or 2 for most home users and y is a number between 0 and 255 which is unique to the network (usually n+1 where n is the last devices name to connect).
Okay, so now that we have the idea of a unique name, electronic devices are quite happy as they can communicate with no issues to each other and know how to find and identify each other.  However, I don't know about you, but I'd never remember that 23.10.153.42 is http://developer.qualcomm.com or that www.gpxblog.com is 172.217.5.83.  So now we need to understand how to make the humans happy.  We also need to make sure that the name the humans use and understand doesn't change, even if the remote machine we want our local machine talk to changes their name (they might move networks, happens all the time).  This is especially true for mobile devices that might change networks between runs of apps or between physical locations.  Chances of using a non changing IP address is not that great.  So let's talk about names one more time and identify another server that keeps track of our human understandable name and relates it to the computer understood IP address.  This is the duty of the Named Server.  Something you can try is to type the console command   "nslookup developer.qualcomm.com" and you should see 172.217.5.83 or something similar if the routing changed.  That query told the machine to resolve the human name developer.qualcomm.com and tell you what the computer address is for it.  Cool huh!
So this naming thing is great, we should take advantage of it, however, one problem.  Most name servers don't have to deal with a lot of changes to IPs the reasoning for this is a physical server doesn't change the way it gets to the internet that often.  So a human created the name and related to a non changing (static) ip.  In mobile, we expect the change to happen often and it'd be great if we didn't have to configure the IP address human name lookup along the way.
This common problem actually has a great solution, the Name Server itself is still a wonderful mechanism but why does it have to be a static server?  Well there's a nice optimization that we can take advantage of.  In normal communication, when our devices go out to query a Name Server for a human to machine name lookup, it would be horrible if we required the local machine to go lookup that same name and resolve it again.  So this information is cached, or simply stored locally.  This way, if you asked your browser to go to gpxblog.com today and then again tomorrow, it would only have to query the NS server today to find out that you meant 172.217.5.83 and saves the lookup from having to happen tomorrow.
Let's use this cool optimization to handle our current problem of needing a dynamic name lookup.  What if we could announce ourselves to all nodes able to hear us what our human name is and that those remote machines should put knowledge of us in their local cache.  Now, no one in the network would need a NS server, and we can take advantage of the optimization and current architecture to work exactly as it normally would.
This system is called mDNS or multicast Dynamic Name Server.  Amazing how the names of systems reflect what they do isn't it?  Now this technique has been around for several decades, but marketing departments being what they are, like taking names and rebranding them to muddy the waters of understanding so Apple calls this Bonjour, and they used to call it Rendezvous.  Microsoft just added support for it to Windows 10 as they've been wanting to go down the road of a technique called uPNP to achieve a similar result but not use a standard, or maintain a single understanding of how it works between versions.  So now we have a standard and knowledge of how to use the standard, that's great let's see some code for how to use it!

DNSServiceErrorType err = DNSServiceRegister(&client, 0, kDNSServiceInterfaceIndexAny, 0, "_gpx._tcp", 0, 0, htons(Port), 5, "\x04Moof", reg_reply, 0);
if (err != kDNSServiceErr_NoError) {
    ERROR("DNSServiceRegister failed with %d", err);
    return false;
}

 First, we want to announce our presence to the world so we tell our local device about our service capabilities.  Next we are going to setup our communication to all other devices with this code:

int sockfd = DNSServiceRefSockFD(client);

int dns_sd_fd = client ? DNSServiceRefSockFD(client) : -1;

int nfds = dns_sd_fd + 1;
fd_set readfds;
struct timeval tv;
int result;

while (!stopNow)
{
    FD_ZERO(&readfds);
    if (client) FD_SET(dns_sd_fd, &readfds);
    tv.tv_sec = timeOut;
    tv.tv_usec = 0;

    result = select(nfds, &readfds, (fd_set*)NULL, (fd_set*)NULL, &tv);
    if (result > 0)
    {
        DNSServiceErrorType locErr = kDNSServiceErr_NoError;
        if (client && FD_ISSET(dns_sd_fd, &readfds))
            locErr = DNSServiceProcessResult(client);
        if (locErr)
        {
            ERROR("DNSServiceProcessResult returned %d\n", locErr);
            stopNow = 1;
        }
    }
    else if (result == 0) {
        ERROR("select timed out");
        stopNow = 1;
    }
    else    {
        VERBOSE("select() returned %d errno %d %s\n", result, errno, strerror(errno));
        if (errno != EINTR) stopNow = 1;
    }
}

Now what's happening here is we're setting up and receiving any replies so our service can now tell other devices about how to talk to us.  Please note that this is an infinite loop so should happen in a separate thread.
Next we need to think about the client:

DNSServiceErrorType err = DNSServiceBrowse(&client, 0, 0, "Tester", "_gpx._tcp", browse_reply, this);

 What this does is tell our device to browse the network for the _gpx service.  This will give us a list and when we get a reply, we know how to deal with it as this is what browse_reply function looks like:

static void DNSSD_API browse_reply(DNSServiceRef client, DNSServiceFlags flags, uint32_t intIndex, DNSServiceErrorType errorCode, const char *name, const char *type, const char *domain, void *data) {
    if (errorCode != kDNSServiceErr_NoError) {
        Network *caller = (Network *) data;
        auto services = caller->ListAvailableServices();
        for (auto service : services) {
            if (service) {
                if (strcmp(service->hostName, name) == 0) {
                    return;
                }
            }
        }
        DNSServiceErrorType err = DNSServiceResolve(&client, flags, intIndex, name, type, domain,
                                                    resolve_reply, data);
        if (err != kDNSServiceErr_NoError) {
            ERROR("couldn't resolveService: %d", err);
        }
    }
}

Here we are with knowledge of the name so now we need to query it so here's resolve_reply:

  static void DNSSD_API resolve_reply(DNSServiceRef, DNSServiceFlags, uint32_t, DNSServiceErrorType errorCode, const char *name, const char *, uint16_t port, uint16_t, const unsigned char *, void *data) {
    if (errorCode != kDNSServiceErr_NoError) {
        Network *caller = (Network *) data;
        auto services = caller->ListAvailableServices();
        for (auto service : services) {
            if (service) {
                if (strcmp(service->hostName, name) == 0) {
                    return;
                }
            }
        }
        Service *service = new Service();
        strncpy(service->hostName, name, (strlen(name) > 256) ? 256 : strlen(name));
        service->port = port;
        service->state = NONE;
        service->version = 0;
        caller->AddFoundService(service);
    }

} 

So at this point we know the other's name, the IP address, the port to communicate on and what kind of communication protocol  to use to communicate.  All that's left is start looking at the actual communication so here's a quick and dirty TCP connection:

clientCallback = Callback;
server.sin_addr.s_addr = inet_addr(host);
server.sin_family = AF_INET;
server.sin_port = htons( port );

//Connect to remote server 

if (connect(socket_desc , (struct sockaddr *)&server , sizeof(server)) < 0)
{
    puts("connect error");
    return false;
}
fcntl(socket_desc, F_SETFL, O_NONBLOCK);
clientThread = new std::thread( [&]()
{
    while (true) {
        uint8_t buf[65536];
        int ret = recv(socket_desc, (char *) buf, sizeof(buf), 0);
        if (ret == -1) {
            int err = errno;
            if( err == ENOBUFS || err == EINPROGRESS || err == EINTR || err == EWOULDBLOCK)
                continue;
            close(socket_desc);
            socket_desc = 0;
            break;
        } else if (ret == 0) {
            close(socket_desc);
            socket_desc = 0;
            break;
        } else {
            // all connected, send and receive data here.
        }
        usleep(0);
    }

} );

That's the client now here's the server:

//Prepare the sockaddr_in structureserver.sin_family = AF_INET;
server.sin_addr.s_addr = INADDR_ANY;
server.sin_port = htons( port );

//Bindif( bind(socket_desc,(struct sockaddr *)&server , sizeof(server)) < 0)
{
    puts("bind failed");
}
puts("bind done");

fcntl(socket_desc, F_SETFL, O_NONBLOCK);

//Listen
listen(socket_desc , 3);

//Accept and incoming connectionc = sizeof(struct sockaddr_in);
serverThread = new std::thread([&](){
    while( !isDone )
        {

            new_socket = accept(socket_desc, (struct sockaddr *)&client, &c);
            if(new_socket < 0)
            {
                int err = errno;
                if( err == EWOULDBLOCK )
                    continue;
                else if(err)
                {
                    ERROR("accept failed");
                    break;
                }
            }
            else            {
std::thread connection_handler([&](){
            //Get the socket descriptor            int sock = new_socket;
            ssize_t read_size = 0;
            uint8_t client_message[2000];

            //Receive a message from client            while( true )
            {
                while(!sendQueue.empty())
                {
                    auto dataToSend = sendQueue.front();
                    StreamType streamType;
                    serialize(dataToSend, streamType);
                    write(sock, streamType.data(), streamType.size());
                    sendQueue.pop();
                }
                read_size = recv(sock , client_message , 2000 , 0);
                if(read_size == -1)
                    break;
                StreamType APImsg(&client_message[0], &client_message[0] + read_size);
                if(deserialize<char*>(APImsg) == "quit") {
                    read_size = 0;
                    break;
                }
                usleep(0);
            }

            puts("Client disconnected");
            if(read_size == -1)
            {
                perror("recv failed");
            }
        });
    }

    if (new_socket<0)
    {
        perror("accept failed");
    }

});

There we go, complete communication between any mobile device.  This will work for Android from the NDK, a 410c device running linux or even a drone (yep something really cool is coming up soon!)  I'll revisit how all of this works in a future post, for now I wanted to keep everything high level and end with a quick demo.  This will allow everyone to follow along with the future explanation for all this code dump.
So until next time.