I’ve Been an IdIoT

From time to time we have to be the family’s IT support. It comes with the profession. Today I had work to do for Mum and Dad. The CCTV just wasn’t working. They were going on holiday, hadn’t checked the cameras in a while but they weren’t accessible from their phones. They really wanted them back before they went away.

The tell tale signs quickly came apparent after I logged in to the NVR (Network Video Recorder) which is a Dahua model. The camera list read “HACKED”, “CHANGE”, “FIRMWARE”. The issue with the device falling off the network turned out to be that a deliberately faulty IPv6 address had been set which also knocked out IPv4 communication.

Best I can tell they had fallen victim to an attack that appeared back in 2017. The default super user account for this NVR is ‘88888888’ and the default password is ‘888888’. Last time I’d checked out the CCTV I noticed this was in place and I really should have changed this. On Dahua NVRs this super user account should not be accessible from remote. The system should check if the access request had been made from local or remote and – in case of the latter – deny access accordingly. The flaw apparently is that this check isn’t properly done. The article linked also references a Facebook post which suggests that using Dahua’s DDNS also made it easy for the attackers to identify and target vulnerable devices. My parents were indeed using Dahua’s DDNS.

In other words the system was a sitting duck on the internet.

I can only assume someone out there has written a script to look for vulnerable Dahua devices exposed to the internet, log in via the admin account using default or easy to guess passwords and then apply setting changes to forcibly remove these devices off the internet. Malicious but also a public service. The device runs an embedded version of Linux and could have easily ended up part of a botnet if hacked firmware was uploaded.

To resolve the attack I reset the unit to factory defaults, uploaded new firmware to both NVR and cameras then set accounts with strong passwords using the XKCD method. I do not have a high trust of IoT devices but not being particularly happy about going up a ladder to replace the cameras this was the best I could do.

It should be noted that Dahua’s website is diabolical for finding firmware updates. You cannot just search a model number to find the device and associated firmware. I needed to check a load of firmware releases to see which ones were pertinent. I had a few attempts with what I thought was correct firmware but apparently wasn’t. I am thankful that the devices can do checks of the firmware to be installed because the risk of bricking the devices in this scenario is real.

In the industry we all know the lessons and we all recognise when a chain of events start happening and not get stopped then the results can be catastrophic. In the case of my parents I assume someone’s shut off network access deliberately but the reality is that someone could’ve used the CCTV access to determine that nobody was in the house. The cameras point to the drive where their cars are parked after all.

Let’s conclude it by reiterating the standard points:

Default passwords (and arguably account names) aren’t acceptable for an internet facing device. I had an opportunity to change it and I didn’t. Whilst the admin account should not have been accessible remotely it was thanks to a firmware flaw. Changing that password could’ve stopped this attack in progress.
Firmware updates need to be done as often as they become available. Security issues happen. Pobody’s nerfect right? As soon as those vulnerabilities and fixes become apparent they need to be tested and deployed.
Trust is a weakness and no trust is owed to the average IoT device. The industry is notorious for bad security practices and abandoning devices within a short lifecycle. Ideally these devices shouldn’t be left open on the internet but placed behind a VPN where the access can be defended slightly better. That’s not a practical solution for most home networks so there must be an accepted risk.
I will reserve judgement on using a DDNS. The CCTV was registered with a *.dahuaddns.com address which I suppose makes it pretty easy to focus an attack if you can narrow it down to specific devices or a single manufacturer. I would suggest using a static IP instead but my parent’s ISP does not offer this. Comment below if you have anything to offer on this as I don’t know how much safer you are with a custom DNS.

I’m A Mechanical Man

This week I was supposed to be on holiday in Cornwall eating pasties and drinking Korev beer. Sadly COVID-19 has put a stop to that for now. Not to worry because Command & Conquer Remastered was released last week and I decided it would be a great way to fill some of the time off I have from work.

I first played Command & Conquer: Red Alert on my cousin’s PlayStation. It was the first strategy game I had ever played and I was hooked despite the pad being absolutely terrible for these kind of games. I then got hold of Command & Conquer for the PC, then Red Alert and the expansions, then everything up to the disappointing C&C 4 back in 2010 which was the last entry in the series. Needless to say Real-time Strategy (RTS) games have been my favourite and C&C my most beloved game series of them all. I’ve played a lot of the greats: Age of Empires, StarCraft, Total Annihilation, Total War, Civilization, Dawn of War. Not one of them in my opinion has ever come close to the classic blend of “thrills, spills and kills” (plus Frank Klepacki’s top notch soundtracks and the B-Movie FMVs of course).

I was very excited back when C&C Remastered was first announced and I’ve been following its development fervently. EA sought assistance from the original Westwood team that went on to establish Petroglyph and also Lemon Sky Studios who have leant their talents to other remasters in the art department. Frank Klepacki provided the remastered soundtrack to both Tiberian Dawn and Red Alert as well as joining forces with live act The Tiberian Sons to add their cover tracks in. The only slight disappointment has been that the source for the FMVs have been lost so AI enhanced upscales have been used and also LAN play has been held back for topical reasons.

And now it’s finally here. From the intro cinematic both of these remastered games speak volumes about the effort that has gone into the game. Both the classic installation sequences were re-done as an intro FMV. These really helped to set the tone for the C&C games and it’s an incredible touch. The upscaled graphics and sound effects are very crisp and look outstanding on my QHD monitor.

Gameplay remains quite the same as the original games. I have been playing through the original Tiberian Dawn GDI missions which I have found quite challenging. They don’t code ’em like they used to. Some missions are straightforward build base and eliminate opponent but some take the form of almost a puzzle. I’ve had to do quite a few restarts due to catastrophic failures but I’ve never felt frustrated.

It’s 1995 again and we’re storming the beach. Did you remember your suncream and sandals this time?

The bonus gallery unlocks a new video each time a mission is completed. These are fascinating to watch as they show each actor directed by Joseph Kucan doing multiple takes to get the perfect cut. A lot of work went into the original game and it’s great to get a look at the archives.

My most favourite part of the game is pressing the magic spacebar key. This switches the graphics between the upscaled assets and the original graphics. As a long time C&C fan I really love hitting the key as the action heats up. It really brings back the memories.

EA have also released the source code for the game DLLs on GitHub so we hopefully will see the mod scene for the game reinvigorated. Time will tell.

So overall a very pleasing release for C&C fans that I will be playing online for some time.

This Week in SQL Server

Actually not a lot. The world is on a stop but I’m still working, running, playing.

So far I’ve done a heck of a lotta jobs around the house because it’s not like we can go many places in the UK at the moment. I have completed an introductory course on Infor OS and I’ve also started one on PowerShell which is going to be useful for all the setup tasks we have to undertake.

Hopefully back soon. Looking forward to open water swimming.

Stay safe.

This Week in SQL Server

The big news this week in SQL Server is the seemingly un-notable release of SQL Server Management Studio 18.5. Scanning the release notes there’s a few new features added but no black theme to the chagrin of many in the SQL Server community it seems. There’s also quite a lot of fixes in this release which is always welcomed however I don’t think that the annoying multi-monitor bug that leaves the query window blank has been fixed as of yet.

If you’re running SQL Server 2017 anywhere there’s also Cumulative Update 20 to test in UAT.

For me this week I got the opportunity to have a self-training day for the company I work for and experiment with (the bizarrely named I must say) Infor OS. I’m not quite there yet with the install but I did decide on creating a Windows Server Core based domain controller and SQL Server to test these on my company issued laptop. I have never really tried server core and I’ve never been asked about it in nearly 4 years of consulting either. It wasn’t as difficult to work with as I first thought and it makes a whole lot of sense to have many cut down Operating System Environments (OSE in Microsoft speak) supporting a single application or role each as opposed to a fully loaded OSE running absolutely everything. It also surely must have been easier on my 4 core, 16GB Laptop as well given that I ended up with 4 Virtual Machines to work on.

The Current Situation

This week marks the end of the third week of isolation at home for me. As our technical work is primarily remote anyway we began this practice 1 week before the UK was put into lockdown. Now that situation is government mandated that means both myself and my colleagues are at home until UK.gov say so.

Not all people have the luxury of staying at home, getting paid and keeping out of harm’s way. I’d like to thank people in my local community for keeping me running during the lockdown: Haigh’s Farm Shop, the three local Co-Op stores in Mirfield, the local Hermes delivery driver and I’m sure many more to come. It’s far too easy to take people for granted and I hope that at least one positive outcome of this lockdown is that we learn to appreciate what these people do for the community.

In terms of work nothing’s really slowed down for me as of yet. I still have upcoming projects to plan, projects going live, live-running systems that need some technical assistance. Should that dry up the plan will be to do some level of self-training to top up our skills. I haven’t had the time to even think this through yet but I suppose brushing up on SQL Server, T-SQL and maybe even learn some Python for fun in my spare time.

It has certainly been interesting watching the sudden adoption of “WFH” (Working From Home) practices. There will of course be long-term ramifications of the COVID-19 pandemic and not every change inflicted will be positive. I’ve certainly not missed the mass-migration that is the 9-5 rush hour and hopefully new working practices can begin the long and arduous process of finally killing that off. Surely there will be many environmental, social, health and economic reasons for that.

Stay safe and keep observing lockdown.

This Week In SQL Server

Oddly enough nothing much to report on this week. It’s been mostly about planning, fixing and patching. Nothing wrong with that every so often.

Of note Cumulative Update 2 for SQL Server 2019 was posted this week with a notably long list of fixes. Not had any deployments of SQL Server 2019 so far but looking forward to doing so. The memory optimised TempDB feature will likely be extremely beneficial for the applications that I deploy and my clients will surely benefit greatly for it.

Where Did 2019 Go?!

It’s been a busy few (many) months at my consultancy job handling the influx of work that has come in. This is thanks to the retirement of Windows Server 2008 & 2008 R2 and also Windows 7. All of those releases were really solid ones (for Windows 7 perhaps the last great Microsoft desktop OS?) and even though they have been surpassed they were very capable OSes for their time.

In saying that it’s most definitely time to move away from them now that extended support has run out. A lot of my time recently has been an effort to get clients from the legacy OS based builds onto something newer. For some clients that means upgrades of the business applications as well as new server builds, for others patching their current applications before moving it to a new server and for the fortunate few just a patch to install. Things like this exemplify the reasons why it’s important in IT to maintain systems and plan for end of support. There is arguably more cost incurred by doing nothing then having to do essentially a project in order to end up with a supported set of OS and applications versus constant planned maintenance.

For the next few months we are concentrating on implementing a new version of the business intelligence tool we resell. It has moved from a direct database connected Excel add-in to one that communicates to a server over HTTP/S. There are many pros and cons to this change and we continue to explore these as we implement with our clients. This will surely keep us busy for a while.

This week in SQL Server largely was the release of Cumulative Update 19 for SQL Server 2017 which I am sure will find its way to a UAT or support server near me soon.

SQL Server Updates

Updating SQL Server is a topic that infrequently gets sent to the support desk which is surprising given the importance of the applications that depend on a SQL database.

The questions usually being:

Is it safe to apply a SQL Server update?
Are you responsible for the update or are we?
What SQL Server updates are certified/supported with our software?

As of SQL Server 2017 the service pack is no more and the cumulative update now has the same level of validation as what the service packs used to have. This simplifies the update process somewhat. Generally it is safe to install these CUs and Microsoft even encourages you to do so on a “proactive” basis.

Installing an update to SQL Server should always be done with a get-out plan in case something goes wrong during the install. That means a properly tested backup and recovery plan.

I have a default rule that when I approach a job and the client asks me to install SQL Server for them I always apply the latest available cumulative update for testing. After that point the client is handed the responsibility for maintenance of SQL Server including the updates.

Most clients I’ve worked on do not patch SQL Server however most have unknowingly applied a SQL update through some form of Windows update mechanism. It’s not always clear if this is WSUS managed or not.

In an ideal world there would be user-acceptance testing done prior to applying SQL Server updates but many businesses do not have the time or facilities to do so. Regardless of the testing done a cumulative update should not break an application and I would always expect the author to be working with the latest service pack or cumulative update during development and support.

In conclusion patching SQL server is something that should be done on a regular basis and an activity that IT administrators need to be confident about handling.

SQL Server 2019 Released

This week Microsoft released SQL Server 2019 RTM.

I’ve not had much time to play with it so far. The headline new feature of SQL Server 2019 appears to be support for Big Data clusters which is not something any of my clients have asked about. However I still think they’d benefit from the in-memory tempdb and some new performance tweaks so I’m keen to check these new features out and see for myself.

I did have issues upgrading from SQL Server 2017 on my desktop PC. I had a Developer edition instance that was previously upgraded from SQL Server 2016. The 2019 installer was failing on the Client Connectivity SDK. I ended up removing as much as I could but was unable to clear away the SQL Server 2016 Setup files. To resolve this I did an install of SQL Server 2016 again and uninstalled everything. This then cleared away the troublesome setup files and I was able to install SQL Server 2017 and SQL Server Management Studio 18.4. If anything this reinforces the best practice of not upgrading SQL Server instances but installing as new.

Some Things I Never Knew In SQL Server

Over the course of the last couple of months I’ve come across some things in SQL Server that I’ve never really thought about.

“..” implies the default schema for A user.

If you have a user in a database with say a default schema of “Production” writing this:

SELECT ProductCode, ProductName, ProductColour
FROM CompanyDB..Products

Actually implies the Production schema.

The TempDB can really cause problems if it’s not configured correctly.

More of a reminder this one. I’ve recently worked on a client who was having some performance issues with a reporting tool. When they ran extracts from a SQL database it wasn’t very fast. I tried it myself and agreed it wasn’t performing as well as it should.

The reporting tool is heavily driven by cursors but let’s not get started on those. I looked in Activity Monitor and noticed quite a lot of contention on the TempDB. A further investigation revealed that there was a total of 1 TempDB for a Octa-core server.

After adding a further 7 TempDB row and log files we found that the reporting tool ran much much quicker and the contention was gone.

You can alias without using “As”

This one probably isn’t “best practice” but you can write:

SELECT C.CustomerID, C.Name, C.Address1, C.City
FROM Customers AS C

As:

SELECT C.CustomerID, C.Name, C.Address1, C.City
FROM Customers C

I don’t like doing this because to me that’s breaking a general principle of keeping script readable plus the reason I’ve never known about this is because all the T-SQL books and resources I’ve read have NEVER mentioned this is correct syntax so they must also consider it a no-no.

You stand a good chance of recovery if your server doesn’t boot but you’ve still got the SQL data.

I recently helped a client that lost a SQL data warehouse server to a Windows update that somehow caused it to not boot. They managed to pull all the files off the volumes so we had all the user databases as well as master and msdb databases.

The client was on a tight budget so we raced against time to get the server going as quick as possible. I was surprised to see that simply dropping in the master and msdb databases over the ones installed on the new server (shut the SQL Server instance down first) and putting the user databases at the original location restored pretty much all settings and data.

Of course you’ll run into problems if there was some encryption in use and you’ll need to use the same SQL Server release but other than that it proved a seemingly OKish way to restore the client’s data warehouse quickly.