Most of you know my day job, for those that don’t, my day job is installing or repairing main-street business internet connections, either on the premises, in the local exchanges, in data centres or in the network operation centres.
I got a pocket full of ID saying I am from all the big players, depending on the job and site and who I am representing.
When I say “business” I don’t just mean high street shops, I do military bases, traffic lights, police stations, water works, schools, children’s homes, motor factors, electrical good suppliers, basically fucking everything that is *NOT* a home broadband / internet connection.
I’ve talked here before, extensively, about how flaky and piss poor it all is, but I know y’all don’t REALLY get it… you’re not nearly scared enough… so, hold that thought, and hold what I typed above, while I do a little detour…
A detour into documented vs undocumented.
A documented change is a change that is ordered or recorded in documentation, and anyone else who has access to that documentation is therefore aware of it… eg “I have replaced the twin 30 ampere breakers with a single 50 amp breaker” and I did this because your wife’s new mega cooker would trip the 30 amp breakers on full load, the downside is you USED to have one 30 amp breaker on your mains socket ring main, and one on the lighting and ventilation circuit… so you USED to be able to trip the socket ring main one and work safely on the sockets and mains gear in the basement while still having the benefit of mains illumination and ventilation, not any more, now it is all either live, or a dark cave.
If you don’t know when you walk in, or when you specify new kit, like the 40 ampere cooker…
In the day job, it usually concerns IP addresses and ranges, and there are only so many to go around, and no tow devices can have the same IP address…
..or rather, they can….. I could in theory put a box on the backbone and give it the IP address 18.104.22.168, which is Google, so you have, effectively, two separate boxes on the internet, both of which are in effect saying “I’M Spartacus!”
The problem is, when you connect YOUR box to the net, and you want to talk to Spartacus and ask where you can find a webpage that will tell you how to wire up a hifi loudspeaker crossover circuit, how does YOUR computer know which is the real Spartacus?
In reality, because your request has to be ROUTED through so many intermediaries, all of whom know who the real Spartacus is, it’s not much of a problem.
(windows users, press Windows key and R simultaneously, a box will pop up, type “cmd” into the box and press enter, this will bring up the console… type “ping 22.214.171.124” and it will tell you how long 126.96.36.199 took to respond to the ping request, and the TTL time to live for the ping request.. all good fun, now type “pathping 188.8.131.52” and be patient for up to three minutes… this command will list all the noted or hops or boxes in between you and spartacus, and after a delay, how many packets each one of them dropped… very useful windows command that has been in there since the XP days..)
So… in the world of BUSINESS network connectivity, things are different, it’s more like there is the internet, and there is the 10 laptops in your own home on your own LAN, except they don’t talk to each other JUST via the router that sits between your and the internet, as in laptop01 > home router > laptop02, but each request goes VIA the internet, laptop01 > store router > internet > store router > laptop02, and laptop01 and 02 may or may not be in the same physical shop.
So we have sites with 950 physical bricks and mortar addresses, and maybe 85 different things connected at each site, and shit isn’t working at site number 157.
Sometimes, you get in a place where you have to stay on the job until it is fixed, eg until site 157 IS working, which means it can talk to HQ (site 001) and a few others, and so on.
Sometimes, according to the documentation, what you have *should* work, but, it doesn’t… somewhere there is a setting that isn’t what you expect, of course that could be anywhere, not just on the corporate VPN, could be in the ISP / provider’s routing tables… who knows, you don’t and you have no way to find out.
Some times, you can look at it and say OK, they have 950 stores, so I’ll just make an imaginary store number 951… change these settings just so, aha, bingo, it works.. but you DARE NOT DOCUMENT what you have done, because according to the documentation, the documented set of numbers and details should work.
Possibly the reason they don’t is back when the company only had 945 stores, someone thought they would fix a problem by making imaginary store 950, who knows.
Real hilarity ensues when you make a mistake and tell the box in store number 949 that it is Spartacus, not knowing that store number 236 also thinks it is Spartacus, because this is a business connection none of the other stores including HQ at store 001 know which is the real Spartacus, so anything can happen, from traffic getting broken up and routed to both 949 and 236, to what should happen, HQ, store 001 saying “I don’t trust either of you bastards” and kicking both store 949 and 236 off the corporate network until a technician (like me) can get on site to sort it… tens of thousands of dollars an hour down the drain, cash sales only in both those stores for the duration.
Real real hilarity ensures when someone makes store 949 think it is Spartacus, and in fact HQ store 001 is supposed to be Spartacus, you can take all 950 stores offline with that one…
Of course, this is what all the ISO / BS / six sigma and other bullshit is supposed to prevent, everything is documented, but the fact is if you want everything documented, you have to document it, and then design a computer record system around THAT, not what happens, which is to tweak the documented system to that it fits the computer programmers design and code.
Which means there is a LOT of scope for undocumented shit, because if it is undocumented, there is no way you can blame employee payroll #628 for doing it…. which is why employee #628 made fucking sure it wasn’t documented.
Between THAT, and shit just falling through the cracks, eg work order was generated for a telecom guy to turn up on site and install a VDSL/FTTC NTU network termination unit / socket, guy turn up and does this, nobody told him it should not go here with all the other stuff that comes in from the street cabinet, but in the comms room 4 doors down the corridor, and nobody told him to install a VDSL modem as well, or even if they did tell him, nobody made sure that there was actually a spare mains power plug anywhere near the new install site to power the VDSL/FTTC modem… you get the picture, between these falling between the cracks sorts of issues, and the undocumented shit issues, you just about accounted for 98% of all issues.
Thing is, my day job EXISTS to put out the fires caused by this 98% of issues, I am a full time fireman, and I am always putting out fires in the IT infrastructure, because there is always something somewhere on fire, and there is a never ending supply of new fires being started because of shit falling between cracks and undocumented shit..
So now it has become the new norm… people in the industry and joining the industry have never known anything except a hideously complex ghetto with guys like me running all over it on our insectile feet putting out fires… of which there are always thousands…
I mean, I get on site, there is flammable material everywhere, and you are all sat around smoking and flicking your dog ends everywhere, and you actually expect me to give a fuck when YOU clearly don’t?
In the past THREE FUCKING YEARS, do you know how many sites I have gone to where there was a printed network map fixed to the cabinet door?
It’s a smaller number than the number of sites I have gone to with a clean and tidy and professionally racked and cabled cab, and that is about three in total.. the first number is zero by the way.
NORMAL is getting on site with a new router, because, you know, the problem with your old EFM service wasn’t all of the above, it is that it wasn’t fibre, so we will upgrayeed all this shit for y’all, and significant expense of course, it’s an exercise in upselling, not remedy… so there I am, fuck all documentation apart from a postal address and and end user name and a telco name, it would be nice to know the circuit ID I’m supposed to be connecting to, or the LAN switch port, but fuck that, the cisco command “sh int desc” *might* tell me, assuming I know the login and pass, assuming the monkey that wrote the config bothered to put such REM statements in, assuming that even when they have it is more informative than “GE0/1 LAN – GE 0/0 WAN”
When I work it all out it still doesn’t work, because you have an ADVA box that is providing 100 meg fibre bearer and a Cisco box that someone has written in the config for GE0/1 “speed 100” which won’t work, because the Cisco box is talking to a non-Cisco box, so it just sits there with a dead circuit, so if I have the login and pass I change it to “speed auto” and bingo it just fucking works, write memory, another undocumented change, not a biggie as these things go.. but another brick in the wall.
Fuck it, I get my forms signed and I’m outta there.
And if y’all think there is ANY part of modern civilised life that does not just stop fucking dead when the internet connections for it go down, y’all just haven’t been paying fucking attention.
And baby, when push comes to shove, it’s the arms of the state that will get connectivity second, the multinationals like google and fuckbook and the banks and major supermarket chains first, infrastructure like traffic lights and water plants third, and as for your home broadband, you’ll come last, and by that I mean y’all will get FUCKING limited connectivity on an ad-hoc basis to one or two proxy services only… as in AOL walled garden days, port 80 http traffic and throttled at that, or sweet fuck all.