Is there a good resource or book for learning about some of the details of how webservers work?
For example if I want an IP address on a intranet to be a webpage that people on that intranet can go to... how would I set that up from scratch. Let's say I have a machine with a static IP on the local net... (but what I really also need to understand is how a static IP is established locally, a DNS?)
Maybe the dream book or resource doesn't exist. But I ask anyway.
@futurebird maybe this page might be instructive on just how simple a "web server" can be. its a single-line shell script (serving out some html). but also explores some other examples.
@futurebird The answer depends on how “from scratch” you want. Here’s an overview. I don’t know if there’s a single book which covers all of this, but there are numerous books covering chunks of it. I’d be happy to expand on any part. I’ve done the “from scratch” down to writing my own network stack, though not making my own processors or network interfaces to run it.
An IP address is more properly called an IP number. It’s just a number you set on a network interface. For IPv4, this number can be chosen for you via Automatic Private IP Addressing (APIPA; 169.254/16) or DHCP, or you can set it manually. IPv6 adds a few methods like StateLess Address AutoConfiguration (SLAAC).
DNS is only needed if you want to use a name to refer to the server instead of the IP number. It’s purely because people remember pronounceable names better than they remember numbers.
IP numbers by themselves can refer to a machine, but they don’t let you know what that machine does. For that, you need to use protocols which run on top of IP such as TCP or UDP. TCP and UDP both use a number called a port to further determine what on the machine you want to interact with. If an IP number is like a street address, the port number is like a tenant name and apartment number inside the building. Web servers use TCP, so I will focus there.
A web server application (typically called a dæmon or a service) runs on the machine and tells the machine to accept connection requests on one or more TCP port numbers (typically 80 and/or 443). When the OS receives traffic on that port, it negotiates a connection with the other machine sending the traffic. This connection establishment process involves a handshake in which both ends pick a random “sequence number”. As the ends exchange traffic, the sequence numbers increase. Both ends send periodic acknowledgements of the most recent sequence number they received from the peer so they can determine if anything is lost in transit.
Once the connection is established, the other machine starts sending web traffic (typically an HTTP request), which the OS running the web service delivers to the web service. The service then decides how to respond based on its configuration. This details are down to the specific web service you use, such as Apache, Lighttpd, nginx, and so on.
One of the ways a web service can respond to a request is to make its own request to some other web server. This is what a web proxy (sometimes called a “gateway”) does. If you ever see the error “504 Gateway timeout”, this is the part which failed.
One bit that I don't think you've been told about is 'nat port forwarding'
If you have a computer that you want to be a web server on your local network it will have a local IP address.
Your router/modem will have* a public IP address. That means that the internet won't be able to talk to the computer on your local network. You can't set up DNS to your local address (or at least not for people outside your network) and if you set up DNS to point to your router the request won't automatically go to your web server.
But you can set up a forwarding rule on the router to take any traffic on particular ports and forward it to your server and it will work.
You can kind of think of NAT as a mailroom in a big office block. External post goes to the mail room and it then delivers the mail to the correct place internally. The postal service only needs to know about the physical location of the mail room.
@futurebird For setting up a local website on an Intranet I can't think of a networking book that wouldn't be massively overkill, though I can still recommend something if you want. Zimmy's descriptions of IP/Ethernet/DNS are all correct but also largely concerned with serving pages over the Internet (although still more concisely explained than I could manage, and definitely worth the read).
I personally would break down the problem into three parts, the web "server" itself, the content you're hosting, and then how it's reached by other users.
For the web server as mentioned this will be software running on your computer, I would recommend Apache. Reason being it uses directories and .htaccess files to serve content, which I find easier to grok. Nginx would be a likely 2nd choice if you were curious. Through the .htaccess configuration you can share a directory to your intranet. Apache itself will also need to be configured, essentially your device needs to say that there's a web page to be hosted, and Apache needs to know to host it where your device says. Mac is not my area of expertise but I found this overview that touches on the important points: https://discussions.apple.com/docs/DOC-250004361
The content itself will need to set up in such a way that is readable by the web server. For a static website this would likely just be an HTML file, CSS file, and possibly a JavaScript file in a folder with the .htaccess file for the web server.
Lastly is how you share it with others. If you have configured your virtual host in Apache correctly (from the guide above) then those local to you should be able to see it. The only issue you may find is that you get a warning that the site is considered unsafe, because there's no SSL certificate in order to serve the web page over an encrypted connection. You could generate your own certificate to bypass this but for personal projects I never bother, personally I've found that Firefox trusts me to continue whereas Chrome just forbids it.
@futurebird if it’s only intranet and only Macs, you might be able to get away with not doing anything for the domains thanks to Bonjour (self-assigned computer-name.local domain names, I use this regularly to test mobile web stuff running on my laptop).
Alternatively, you can configure a public domain to point to the internal IP, assuming it’s static (I also have this locally for services running on non-Macs.) if the domain/IPs must stay secret, it requires an internal DNS server as well.
@futurebird Just noting that I’m glad you asked this, cos this is a fascinating thread with people posting a lot of interesting sources and comments*. I hope you get a good answer out of it!
MacOS used to have a built-in web server but Apple took that out a bunch of versions back.
*a lot answers covering anything I would have said and more breadth, thus just the above. 😃
@futurebird Thank you for this great thread. Apologies if this has already been posted (looked, didn't see it) but there's a great list of one-liner mini-webservers here: https://gist.github.com/willurd/5720255 - any one of these may be a good jumping off point for your own further explorations in the language of your choice, plus it's a useful tool to have around when you just want to serve a local directory quickly and temporarily without faffing around in nginx or or Apache.
If you want to just understand what a web server does, people have dropped pointers here already.
One person that I deeply admire is W. Richard Stevens, who unfortunately died much too early.
He wrote https://www.amazon.com/-/de/dp/0321336313. I do not have that edition, I only have the first edition, and I hope they did not mess with his clarity in later editions.
Specifically the first volume of TCP/IP Illustrated takes the spec of TCP/IP and a packet monitor, and then compares the spec …
@futurebird I honestly don't remember where I learned this stuff from because I'm old. It wasn't a book. When I was younger I managed to finagle my way into a small shop IT job and Googled my way from there. Which apparently is not so easy anymore.
The short answer to your question is you need to point devices on your intranet to an internal DNS server that resolves domain names it knows and boots up names it doesn't know to external DNS servers. Which I guess is not really a short answer.
@futurebird I don't have a resource for you, but want to give you a heads up of something you'll run into, if you haven't already: there is a whole "stack" of technologies that take us from cables to networks to IPs to the web. They are made to layer and be isolated, but also to build on each other. Getting a computer to serve a webpage on a local network involves knowing a little bit about somewhere between a few and several of these layers.
@futurebird static IP live in the router. Each wifi / ethernet adapter have a mac address, router have a list of mac address and IP it should be attributed. From there, either the server request an IP using dhcp and it will happen to be the static one or the server just use the static IP and hope things will be fine.
A DNS let you map a human readable name to an IP. Like "phonebook.local => 10.10.144.37".
@futurebird Do you want to do this yourself? Do you want to start from scratch? Do you want teaching material?
An http1.1 server is actually a ridiculously easy program to write. One needs only to know a dab of multi-threading for the sockets, and lots of String manipulation. Won't be fancy, will work.
Of course, most programming languages have deluxe cool libraries, and they're even simpler to use.
I want to understand it with more depth. For example I had a great time playing with Django and getting little webpages running with python, but there were a lot of details I didn't understand. Or even is Django like a small basic version of Apache? I assume it is I had to turn off one to run the other or else put them on different sockets.
There is a big difference between blundering around making things happen and really understanding what's happening.
@futurebird Gotcha. I wrote a crude http server (kotlin) from scratch a couple of years back. Didn't get terribly far, but I learned a lot. There's a guy who wrote one from scratch, and had just couple of blog articles, and they really helped me. I'm looking for them, but not luck, yet.
I will keep looking. The kotlin aspect is irrelevant, really.
Or even is Django like a small basic version of Apache?
no, they are entirely different things. apache is a generic http server that serves static files (serving files as-is), forward connections to secondary servers, and is also a php runtime.
django is a web framework, and that means you have to write a program on top of it. this program can technically does what apache does, but there is no point to it—you're better off just using apache (or nginx).
a python's equivalent to apache would be python -m http.server, which is indeed a very basic web server, way more basic than apache and only serve static files from one directory without much configs.
I assume it is I had to turn off one to run the other or else put them on different sockets.
yes, apache and the server you implement using django have to run on different ports. otherwise, when you request, http://localhost:<the port> , how could it know which server handles the request? it is common, however, to run both apache (or nginx) and other servers, with the generic server being a gateway to forward requests to other servers. with this setup, the gateway server (apache/nginx) listens to http/https ports (80/443) and other servers listens to different port, or to their own sockets. we then declare in the apache's config file that if a request matches this pattern, then it should forward the request to that server
@futurebird I cut my teeth on all that when our internal firewall failed and I had to replace it, pfsense and opnsense are stack solutions that accomplish what you're thinking usually off a lil heat-sync box... Local DHCP server + "unbound" for DNS server. From unbound (there are probably alternatives) you can assign domain names to IP addresses (or block requests to certain domain names) DNS middle-person is very powerful
@specter OK That's interesting, so the same software that does DNS blocking might be able to assign local names? Part of what I'm dealing with is the sysadmin, who is a nice guy, but kind of irritated that I want the CS club playing with all this stuff. So, the more I know about what exactly is "easy" the better.
@futurebird what I thought was maybe kinda easy (playing in my homelab after the firewall fire) is assigning an intranet domain name in pf/opnsense and then configuring unbound to save/serve local devices as <hostname>.<intranet>... Since it's local DNS server it doesn't have to be a purchased TLD for the intranet part
@futurebird There's a "to create anything from scratch, first create the universe" answer to that (and I'll see if I can think of any books that provide that), but for an overview of the mechanics, this page isn't bad: https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview
@futurebird Grab a copy of the Internet guide for dummies. It’s a great read and clearly explains all the concepts. Note internet/intranet use the same technology.
@futurebird
IPs on LAN are allotted automatically by DHCP server running on the router usually. This can also be done manually. You can also assign a static IP to a machine while having DHCP enabled. You do this in the router configuration, and that IP will be tied to the MAC address of said machine.
@futurebird@dushman
Yeah, a typical DNS server will do that — give names to IPs that work locally, while forwarding requests for anything else upstream (either to the ISP DNS server, or directly to the DNS servers across the Internet)
It's up to the person setting up the DNS server to make sure that those IP addresses correspond; there isn't any magic matching it with the addresses DHCP hands out
You asked for a book, but I'm going to stretch that a little and offer some lecture notes from a short course I gave years ago. This is a lecture on how web servers work. Maybe there's still some useful information? (Apologies in advance for the terrible presentation.)
I am assuming this is all because you are curious and want to learn the nitty gritty details, or want to teach about it.
If on the other hand, you are just trying to setup a private web server on your LAN to share data, you can probably download a specific Raspberry PI distribution and follow a few page README. Something like Internet in a box maybe:
@futurebird There are a couple of ways a static IP could be assigned in an Intranet.
Usually you get a dynamic IP address from DHCP, a service running on the router you connect to. It identifies a network interface (your network card or wifi) by its MAC address, then assigns an IP address for a certain period of time.
The most reliable way to assign a static IP address is for it to be reserved on the DHCP server, so that MAC always gets the same IP.