IT professional specialising in Microsoft SQL Server, Windows Server and Azure based in West Yorkshire, United Kingdom. Driven to learn, improve and create better IT outcomes for clients. Open water swimming enthusiast and facilitator for Andy's Man Club.
I’m going to let the whole thing settle down before I jump to any conclusions because let’s face it that’s as much exercise as some armchair political pundits have had for the whole crisis. I’ve also just had to peel myself from the ceiling reading a Reddit comment describing databases as “one of the lowest forms of computing”.
Take a deep breath. The UK.gov failed to disclose circa 16,000 cases of COVID-19 allegedly because of a technical problem that turned out to be the Excel spreadsheet they were using surpassed the 1,048,576 row limit in Excel.
No I don’t know either.
I usually steer clear of politics but seen as we have spent around £11 million quid on the initial contact tracing app which didn’t even work I feel like it’s all becoming a bit of a farce over there. Thank goodness Dido Harding is at the head of the new alphabet soup in charge of public health in England otherwise we’d be really in dire straits now.
Like I said before I’m absolutely not jumping to conclusions. I’m getting my exercise elsewhere and I’m coming on leaps and bounds with my sarcasm problem.
From time to time we have to be the family’s IT support. It comes with the profession. Today I had work to do for Mum and Dad. The CCTV just wasn’t working. They were going on holiday, hadn’t checked the cameras in a while but they weren’t accessible from their phones. They really wanted them back before they went away.
The tell tale signs quickly came apparent after I logged in to the NVR (Network Video Recorder) which is a Dahua model. The camera list read “HACKED”, “CHANGE”, “FIRMWARE”. The issue with the device falling off the network turned out to be that a deliberately faulty IPv6 address had been set which also knocked out IPv4 communication.
Best I can tell they had fallen victim to an attack that appeared back in 2017. The default super user account for this NVR is ‘88888888’ and the default password is ‘888888’. Last time I’d checked out the CCTV I noticed this was in place and I really should have changed this. On Dahua NVRs this super user account should not be accessible from remote. The system should check if the access request had been made from local or remote and – in case of the latter – deny access accordingly. The flaw apparently is that this check isn’t properly done. The article linked also references a Facebook post which suggests that using Dahua’s DDNS also made it easy for the attackers to identify and target vulnerable devices. My parents were indeed using Dahua’s DDNS.
In other words the system was a sitting duck on the internet.
I can only assume someone out there has written a script to look for vulnerable Dahua devices exposed to the internet, log in via the admin account using default or easy to guess passwords and then apply setting changes to forcibly remove these devices off the internet. Malicious but also a public service. The device runs an embedded version of Linux and could have easily ended up part of a botnet if hacked firmware was uploaded.
To resolve the attack I reset the unit to factory defaults, uploaded new firmware to both NVR and cameras then set accounts with strong passwords using the XKCD method. I do not have a high trust of IoT devices but not being particularly happy about going up a ladder to replace the cameras this was the best I could do.
It should be noted that Dahua’s website is diabolical for finding firmware updates. You cannot just search a model number to find the device and associated firmware. I needed to check a load of firmware releases to see which ones were pertinent. I had a few attempts with what I thought was correct firmware but apparently wasn’t. I am thankful that the devices can do checks of the firmware to be installed because the risk of bricking the devices in this scenario is real.
In the industry we all know the lessons and we all recognise when a chain of events start happening and not get stopped then the results can be catastrophic. In the case of my parents I assume someone’s shut off network access deliberately but the reality is that someone could’ve used the CCTV access to determine that nobody was in the house. The cameras point to the drive where their cars are parked after all.
Let’s conclude it by reiterating the standard points:
Default passwords (and arguably account names) aren’t acceptable for an internet facing device. I had an opportunity to change it and I didn’t. Whilst the admin account should not have been accessible remotely it was thanks to a firmware flaw. Changing that password could’ve stopped this attack in progress.
Firmware updates need to be done as often as they become available. Security issues happen. Pobody’s nerfect right? As soon as those vulnerabilities and fixes become apparent they need to be tested and deployed.
Trust is a weakness and no trust is owed to the average IoT device. The industry is notorious for bad security practices and abandoning devices within a short lifecycle. Ideally these devices shouldn’t be left open on the internet but placed behind a VPN where the access can be defended slightly better. That’s not a practical solution for most home networks so there must be an accepted risk.
I will reserve judgement on using a DDNS. The CCTV was registered with a *.dahuaddns.com address which I suppose makes it pretty easy to focus an attack if you can narrow it down to specific devices or a single manufacturer. I would suggest using a static IP instead but my parent’s ISP does not offer this. Comment below if you have anything to offer on this as I don’t know how much safer you are with a custom DNS.
This week I was supposed to be on holiday in Cornwall eating pasties and drinking Korev beer. Sadly COVID-19 has put a stop to that for now. Not to worry because Command & Conquer Remastered was released last week and I decided it would be a great way to fill some of the time off I have from work.
I first played Command & Conquer: Red Alert on my cousin’s PlayStation. It was the first strategy game I had ever played and I was hooked despite the pad being absolutely terrible for these kind of games. I then got hold of Command & Conquer for the PC, then Red Alert and the expansions, then everything up to the disappointing C&C 4 back in 2010 which was the last entry in the series. Needless to say Real-time Strategy (RTS) games have been my favourite and C&C my most beloved game series of them all. I’ve played a lot of the greats: Age of Empires, StarCraft, Total Annihilation, Total War, Civilization, Dawn of War. Not one of them in my opinion has ever come close to the classic blend of “thrills, spills and kills” (plus Frank Klepacki’s top notch soundtracks and the B-Movie FMVs of course).
I was very excited back when C&C Remastered was first announced and I’ve been following its development fervently. EA sought assistance from the original Westwood team that went on to establish Petroglyph and also Lemon Sky Studios who have leant their talents to other remasters in the art department. Frank Klepacki provided the remastered soundtrack to both Tiberian Dawn and Red Alert as well as joining forces with live act The Tiberian Sons to add their cover tracks in. The only slight disappointment has been that the source for the FMVs have been lost so AI enhanced upscales have been used and also LAN play has been held back for topical reasons.
And now it’s finally here. From the intro cinematic both of these remastered games speak volumes about the effort that has gone into the game. Both the classic installation sequences were re-done as an intro FMV. These really helped to set the tone for the C&C games and it’s an incredible touch. The upscaled graphics and sound effects are very crisp and look outstanding on my QHD monitor.
Gameplay remains quite the same as the original games. I have been playing through the original Tiberian Dawn GDI missions which I have found quite challenging. They don’t code ’em like they used to. Some missions are straightforward build base and eliminate opponent but some take the form of almost a puzzle. I’ve had to do quite a few restarts due to catastrophic failures but I’ve never felt frustrated.
It’s 1995 again and we’re storming the beach. Did you remember your suncream and sandals this time?
The bonus gallery unlocks a new video each time a mission is completed. These are fascinating to watch as they show each actor directed by Joseph Kucan doing multiple takes to get the perfect cut. A lot of work went into the original game and it’s great to get a look at the archives.
My most favourite part of the game is pressing the magic spacebar key. This switches the graphics between the upscaled assets and the original graphics. As a long time C&C fan I really love hitting the key as the action heats up. It really brings back the memories.
EA have also released the source code for the game DLLs on GitHub so we hopefully will see the mod scene for the game reinvigorated. Time will tell.
So overall a very pleasing release for C&C fans that I will be playing online for some time.
Actually not a lot. The world is on a stop but I’m still working, running, playing.
So far I’ve done a heck of a lotta jobs around the house because it’s not like we can go many places in the UK at the moment. I have completed an introductory course on Infor OS and I’ve also started one on PowerShell which is going to be useful for all the setup tasks we have to undertake.
Hopefully back soon. Looking forward to open water swimming.
The big news this week in SQL Server is the seemingly un-notable release of SQL Server Management Studio 18.5. Scanning the release notes there’s a few new features added but no black theme to the chagrin of many in the SQL Server community it seems. There’s also quite a lot of fixes in this release which is always welcomed however I don’t think that the annoying multi-monitor bug that leaves the query window blank has been fixed as of yet.
If you’re running SQL Server 2017 anywhere there’s also Cumulative Update 20 to test in UAT.
For me this week I got the opportunity to have a self-training day for the company I work for and experiment with (the bizarrely named I must say) Infor OS. I’m not quite there yet with the install but I did decide on creating a Windows Server Core based domain controller and SQL Server to test these on my company issued laptop. I have never really tried server core and I’ve never been asked about it in nearly 4 years of consulting either. It wasn’t as difficult to work with as I first thought and it makes a whole lot of sense to have many cut down Operating System Environments (OSE in Microsoft speak) supporting a single application or role each as opposed to a fully loaded OSE running absolutely everything. It also surely must have been easier on my 4 core, 16GB Laptop as well given that I ended up with 4 Virtual Machines to work on.
This week marks the end of the third week of isolation at home for me. As our technical work is primarily remote anyway we began this practice 1 week before the UK was put into lockdown. Now that situation is government mandated that means both myself and my colleagues are at home until UK.gov say so.
Not all people have the luxury of staying at home, getting paid and keeping out of harm’s way. I’d like to thank people in my local community for keeping me running during the lockdown: Haigh’s Farm Shop, the three local Co-Op stores in Mirfield, the local Hermes delivery driver and I’m sure many more to come. It’s far too easy to take people for granted and I hope that at least one positive outcome of this lockdown is that we learn to appreciate what these people do for the community.
In terms of work nothing’s really slowed down for me as of yet. I still have upcoming projects to plan, projects going live, live-running systems that need some technical assistance. Should that dry up the plan will be to do some level of self-training to top up our skills. I haven’t had the time to even think this through yet but I suppose brushing up on SQL Server, T-SQL and maybe even learn some Python for fun in my spare time.
It has certainly been interesting watching the sudden adoption of “WFH” (Working From Home) practices. There will of course be long-term ramifications of the COVID-19 pandemic and not every change inflicted will be positive. I’ve certainly not missed the mass-migration that is the 9-5 rush hour and hopefully new working practices can begin the long and arduous process of finally killing that off. Surely there will be many environmental, social, health and economic reasons for that.
Oddly enough nothing much to report on this week. It’s been mostly about planning, fixing and patching. Nothing wrong with that every so often.
Of note Cumulative Update 2 for SQL Server 2019 was posted this week with a notably long list of fixes. Not had any deployments of SQL Server 2019 so far but looking forward to doing so. The memory optimised TempDB feature will likely be extremely beneficial for the applications that I deploy and my clients will surely benefit greatly for it.
It’s been a busy few (many) months at my consultancy job handling the influx of work that has come in. This is thanks to the retirement of Windows Server 2008 & 2008 R2 and also Windows 7. All of those releases were really solid ones (for Windows 7 perhaps the last great Microsoft desktop OS?) and even though they have been surpassed they were very capable OSes for their time.
In saying that it’s most definitely time to move away from them now that extended support has run out. A lot of my time recently has been an effort to get clients from the legacy OS based builds onto something newer. For some clients that means upgrades of the business applications as well as new server builds, for others patching their current applications before moving it to a new server and for the fortunate few just a patch to install. Things like this exemplify the reasons why it’s important in IT to maintain systems and plan for end of support. There is arguably more cost incurred by doing nothing then having to do essentially a project in order to end up with a supported set of OS and applications versus constant planned maintenance.
For the next few months we are concentrating on implementing a new version of the business intelligence tool we resell. It has moved from a direct database connected Excel add-in to one that communicates to a server over HTTP/S. There are many pros and cons to this change and we continue to explore these as we implement with our clients. This will surely keep us busy for a while.
This week in SQL Server largely was the release of Cumulative Update 19 for SQL Server 2017 which I am sure will find its way to a UAT or support server near me soon.
Updating SQL Server is a topic that infrequently gets sent to the support desk which is surprising given the importance of the applications that depend on a SQL database.
The questions usually being:
Is it safe to apply a SQL Server update?
Are you responsible for the update or are we?
What SQL Server updates are certified/supported with our software?
As of SQL Server 2017 the service pack is no more and the cumulative update now has the same level of validation as what the service packs used to have. This simplifies the update process somewhat. Generally it is safe to install these CUs and Microsoft even encourages you to do so on a “proactive” basis.
Installing an update to SQL Server should always be done with a get-out plan in case something goes wrong during the install. That means a properly tested backup and recovery plan.
I have a default rule that when I approach a job and the client asks me to install SQL Server for them I always apply the latest available cumulative update for testing. After that point the client is handed the responsibility for maintenance of SQL Server including the updates.
Most clients I’ve worked on do not patch SQL Server however most have unknowingly applied a SQL update through some form of Windows update mechanism. It’s not always clear if this is WSUS managed or not.
In an ideal world there would be user-acceptance testing done prior to applying SQL Server updates but many businesses do not have the time or facilities to do so. Regardless of the testing done a cumulative update should not break an application and I would always expect the author to be working with the latest service pack or cumulative update during development and support.
In conclusion patching SQL server is something that should be done on a regular basis and an activity that IT administrators need to be confident about handling.
