BD – Hortonworks Data Flow

HDF Masterclass (NiFi)

I attended the Hortonworks Data Flow Masterclass on Thursday this week (18th). It is a very comprehensive, simple to configure, piece of software. It facilitates the rapid configuration of managing data in flight. HDF simplifies the configuration of many types of incoming data flows in different protocols and formats, transforming it, redirecting it, and sending it out to another type of data flow. You basically set up “processor” blocks, apply configurations to those block to handle the data in the way you want. You then add further building blocks to build up the overall data flow that you require. It has many many other features, far to detailed for this blog that make it a excellent tool to investigate if you are having any issues getting your data into a “Data Lake”.

There is a very simple GUI that has drag and drop features that is very easy to use to build complex data flows quickly. For example, Suppose you have an FTP repository location that files are written to and you want those written to HDFS, how do you do that? Now obviously, you can do the FTP to HDFS translation without any effort using Isilon due to its multi-protocol capability, but what if you wanted to do some transformation of the data? What if you wanted to apply some meta data, maybe time-stamp in the file name and then compress the files and then write to HDFS. You can of course script such a simple process but you then you have to support it moving forward and fix issues that occur should the data format change, locations change and so on.

With HDF, this data flow process would be created as followed:

  • You start with an input processor that looks at the remote ftp repository and moves files to the local location via FTP or most likely SFTP.
  • A second attribute process block could be used to rename the file to include the time-stamp and add the required compression method as a meta data object.
  • The output would be directed to a compression processor which carries out the compression of the data based on the the type of compression algorithm object specified in the previous step
  • The output of the above processor would then be directed to an HDFS file output processor writing the file into an Hadoop file system.

These processor block are dragged/dropped onto the page in the GUI and configured with a few mouse clicks. They are connected together by clicking on the source processor and dragging an arrow to the next processor. The above configuration would take just a few minutes to build. It is then started using stop/go buttons and can be tested. Each block shows ingress/egress statistics and you also have very detailed data lineage visibility at every stage of the process. You can look at the actual data anywhere along the process and even replay the data back through the data flow to retry should you find any errors or want to add additional data processing into the path.

Some immediate use cases are apparent straight away.

  • Sure you can write scripts to do any kind of data ingest, transform and output data but those are the boring tasks. Why not offload those tasks to a very simple to use tool that can carry out the process without any scripting with great visibility of what is going on. It has many inbuilt functions to simply add to the outlying error cases that will crop up over time. That allows your staff to get back to the real work of generating business benefits from the data you have captured rather than spending all their time just managing the data flow processes.
  • What if you use a BI tool that charges per TB ingested and although you want to keep all of the data eventually, a lot of the incoming data is noise. Why not filter the noise off into a data lake and only pass the relevant data into your BI tool thereby reducing the per TB ingest costs dramatically.
  • …. many others

I could go on and on about this. A number of my colleagues have seen the product and have all been excited about the possibilities it raises. If you are involved with any kind of data management requiring the collation of data (hopefully you are if you have found these pages) then please take a look at HDF and see what you think. Of course, please feel free to feed back any comments below.

XJ6R – Quick Spray job Part 2

XJ6R – Quick Spray job – Part 2

I managed to get a few short hours in the garage over the weekend when it was not raining outside. I needed it to be dry so that I can put my small compressor outside to feed fresh air to the face mask. I had done some wet sanding to get rid of the worst of the paint runs (from part 1) and of course I had gone through the paint and back to the metal in a couple of places. I re-masked off the various parts and resprayed the inner wing again. Although not as good as I would like, it is passable. Especially as this is the side where the Supercharger goes, along with a few ancillaries so there is not a lot of room left for you to see any bad paint work! I will let it harden for a few more days and give it a light buff and polish. Apologies for the picture quality, trying to show a paint finish is very difficult at the best of times but this is black in a fairly dark part of the garage. Also the phone camera is not that good either.

Inner Wing
Inner Wing

I also made a start at cleaning the engine up a little as it looks quite a mess. I removed some of the metal work that makes up the inter-cooler air feed pipes and washed them in solvent. They do look a lot better but I might have to resort to polishing them or even spraying them. They are aluminum but have a kind of textured finish to them. I guess that there is more exterior surface area meaning that it might allow more cooling from passing air? I was wondering what other people have done with these engines. I did see an E-Type with a fully polished (very shiny) XJR6 engine installed so maybe that’s the extreme.

Engine prior to cleaning
Engine prior to cleaning

It is -2 outside so I will do some more research and planning!

I did a quick search and found a video on youtube of a Supercharged E-Type with the polished pipe work which looks nice, (see around 1:50 in) I am not so keen on the wheels though!


XJ6R – Quick Spray job

XJ6 R engine bay preparation and respray

I had mucked up an earlier attempt at respraying the passenger side of the engine bay. Firstly I had covered part of the inner wing with tape around the VIN plate so that needed redoing and I also scratched the side during the fitting of some of the new items needed in the engine bay. You can see the initial repairs highlighted by the red primer and can also see the new wiring (temporarily laid out).

Side view temporary wiring
Side view temporary wiring
Temporary Engine Bay Wiring
Temporary Engine Bay Wiring

Rather than repair it and then damage it again as I carried on working on fitting the items and the new wiring needed in the engine bay. I also needed to cut a hole as that discussed in the cutting a hole in the bulkhead post.

Respray Preparation

The first step was to remove the new wiring, the fuse box and the newly added modules to allow for the respray. The blue tape masks off the various parts I did not want to remove.

Taped up, prepped for spraying
Taped up, prepped for spraying
Front view, prepped for spraying
Front view, prepped for spraying

Well I tried to respray it and obviously had some setting wrong because no matter how delicate and fast I was, I still ended up with a significant amount of paint runs. I guess I am a better storage  technology guy than a paint sprayer 🙂

I have left it for now. I will carefully wet sand out the paint runs and see what it looks like in a week or so. stopped there as I decided to write this and then look to see if I can find some of the items I need to purchase.

Here is my current list of items to locate and buy next.

  • Brake Master Cylinder and clips to hold on the reservoir
  • Reconditioning kit for the steering rack plus the bellows
  • Front shocks
  • Front suspensions ball joints
  • Alternative fuel pump as the ones in the twin tacks will not provide enough pressure on their own
    • need some more thought regarding the fuel system such as can I use the XJR tank or do I use a surge tank as suggested by Larry Louton on the Jaguar forums page
  • The rubber pipe that connects to the battery tray and the air inlet by the windscreen (BD4654) because mine is split
  • Supercharger Belt for the XKR to stop the tweeting noises

Looks like I need to do a little research rather than watching the paint dry 🙂


XJ6R – Master Cylinder and ODBII

Brake Master Cylinders

I spent some time in the garage over the weekend. Most of it was spent inspecting the Brake Master cylinder for the car. I had a pair of these for the XJ6. One I rebuilt, maybe 15 years ago and put in a drawer and the rusty looking one that was the original still in the car. The rebuilt one looked OK externally but had the wrong rubber fittings on the top where the reservoir should sit. It had the two identical rubber fittings that I believe go to the later versions of the car.  These had a separate reservoir attached to the bulkhead via two pipes. My car has the reservoir fitted directly on top of the master cylinder. I had problems getting the pins out that hold the reservoir onto the master cylinder. I ended up cutting and eventually drilling out one of the pins. I then started to disassemble the cylinders themselves. The original master cylinder was very rusty and took a little persuasion to disassemble. Having removed the first of the two plungers, a quick look inside at the cylinder wall proved that it was beyond repair. I think you can see the rust damaged cylinder walls in the photos.

Original Master Cylinder
Original Master Cylinder
This was the spare, no good either!
This was the spare, no good either!

I then started on the one that had new rubbers fitted years ago to see if that was in a usable state. Unfortunately not! It looks like a new master cylinder is needed. They are no longer available from Jaguar Heritage. A quick look through the suppliers I often use reveals that there are  superseded items available.  Not a cheap item but better safe than sorry!


ODBII Socket

As I could do no more to the braking system, I spent a while reviewing some of the wiring and finally decided where I would place the ODBII connector. It is going to go inside the drawer in the central arm rest, between the driver and passenger seats. Not too difficult to wire up, deciding on where to put it took longer than wiring it 🙂 I was initially looking at somewhere under the dash but the centre console seems a better place. I will no doubt require access to it quite frequently, at least in the initial stages of installing and getting the engine running. I still need to check for clearance under the drawer as there are a couple of large pipes feeding the heating/AC flow to the rear of the car. I have left a fair amount of slack in the wiring to allow a lot of positioning options just in case.

There are just a few connection needed.

OBDII Socket wiring
OBDII Socket wiring

Brown/Red – Box 101 is a fused but constant +12v supply

White/Pink – Circle 5 is a switched +12v when key is in position 2 which is normal running position

White/Orange – Hexagonal 3 is a switched ground signal also when the key is in position 2

Pink and Orange are CAN bus serial data lines

Black is chassis and Black/Pink is logic ground.


I had previously wired/reserved the constant +12v, and the switched +12v using the fuse box I installed into the passenger foot well by the door hinges. I had also implemented a relay to provide the switched ground connection elsewhere so I just tapped a wire off of that feed. The rest were either available to tap into around the J-Gate assembly or running across the area to get to the Instrument PCB. The Instrument PCB is going to be installed vertically on the drivers side of the centre console.

Another step closer….

XJ6R – Bulkhead hole

XJ6R Hole cut out

To get the ECU loom from the engine bay into the passenger foot well I needed to make the LHD steering column hole larger. This hole is used to feed the engine vacuum to the AC unit to drive the various flaps and valves.

Vacuum pipes
Vacuum pipe panel
Strengthening ring (after cut out)
Strengthening ring (after cut out)

There is a thick strengthening ring surrounding the standard hole so I decided to cut around that meaning the hole would be around 60mm in diameter. The plugs on the loom would not fit through the standard hole as they are quite bulky.

ECU Plugs
ECU Plugs
Wooden template
Wooden template notice the bulkhead cutout below it for the other required connections

I had to make up a simple wooden template to enable me to centre a holesaw in the right place. The holesaw, along with some cutting fluid made short work of the sheet metal and I was left with a nice clean cutout. Picture taken after a quick clean up and a bit of paint for protection.

Hole cut out
Hole cut out

I will be using the rubber gaiter/plug to seal up the hole as used on the original XJR.

Gaiter on loom
Gaiter on loom

XJ6R – Electronics layout

Electronic Controllers – where to put them?

XJR Wiring looms

As its minus 3 degrees outside I decided not to go work in the garage so I’m writing up some of the work I did last year. As I already discussed here I am attempting to use the minimum of the electronics from the original XJR without having to install all of the wiring as it was a lot, as you can see to the right! The main controllers I need are the following:

  • Engine Control unit (ECU)
  • Transmission Control unit (TCU)
  • Body Control unit (BCU)
  • Security and Control unit (SCU)
  • Instrument Display unit

Now I obviously have these from the XJR along with all the wiring however however I am not going to use all of the rest of the systems. That will of course cause some errors but I am hoping that I can resolve enough of them to allow the engine to run without going into limp home mode. The basic goal is to make the electronics think that it is still in the original XJR with some issues but not enough to shut the engine down.

For example, I need the SCU plus the module that senses that the key is in the lock to unlock the engine immobiliser system. Well I don’t need the actual lock from the XJR. In fact, all I need is to place the key (or just the RFID chip in the key) inside the coil that was around the key lock for that to signal the system that the key is there and to unlock the engine inhibitor.

Another example is the central locking and the door lock sensor system. I am hoping that I can just tie all the signals to either high or low and trick the system into think all is well.

There are a fair few modules that I need to mount somewhere and a lot of wiring that I need to do. I am putting the modules in the passenger foot well. After a lot of thought and trial fitting I have installed the control units as follows.

  • ECU vertically where a passengers soles would go. I plan to put a false panel across this area covering/protecting the modules.
  • TCU Vertically, to the Right hand side of the foot well
  • BCU Horizontally,  directly under the battery tray between the bulkhead and the round Fan assembly
  • SCU vertically and at an angle between the round Fan assembly and one of the output pipes.
  • I also installed one of the XJR Fuse boxes into the left hand side of the foot well behind the door hinges and powered from the through bulkhead bolts used to distribute DC positive power into the interior.

The layout can be seen in the images below:

Control unit location
Control unit location
Controller Unit locations in footwell
Controller Unit locations in footwell.

Above, you can see the inserted panel (red paint) with the rectangular hole for the through-bulkhead connector. I had to make quite a complex and accurate hole in the bulkhead to get a good fit for the bulkhead connector as used in the XJR. I should have cut it out of the XJR body but I scrapped the body without thinking about it. It was easier to make up a small panel on the bench with the required cut out and then weld the small square panel into the XJ6 than try to do it upside down in the foot well. The circular item above the red painted area with two pipes in it, is the access point for the Vacuum pipes from the AC unit. That is the hole used for the steering column on a LHD car. I will use it to get the ECU plugs and cables from the engine to the ECU. I will actually have to cut out the outer edge of this factory made hole as the plugs are quite big. There is a rubber gaiter on the XJR loom which should make a nice fit into this hole keeping the engine bay fumes out of the interior of the car. having to make these holes is a shame as I really don’t want to make too many changes to the body as part of this modification.

At least I stayed warm this evening. I hope the weather breaks soon as I have not really done anything to the XJ6 since New Years day. I did work on the XKR regarding the tweeting. I have however, earned some brownie points doing quite a few DIY jobs in the house.

XKR – Stop that Tweeting!

There was a strange tweeting/squealing noise emanating from the engine bay on the XKR which started a few months ago. It stopped once the engine got warm. I did mention it at the last service and they changed the Supercharger belt. The nest full of hungry birds returned shortly after the service. I tried using a tube and an old funnel to isolate the noise which I deemed to be the water pump. There was some notoriety with early water pumps so I changed it before Christmas. The noise did stop for a couple of journeys however it returned again! Not needing to use the car for a little while, it sat while I decided what to do next. A friend of mine from the JEC (Thanks Neil), lent me an electronic listening device which has a long tube that you place onto the various parts you suspect are making the noises and listen via a pair of headphones. You could not hear the tweeting but a rumbling noise and some were worse than others. One of the tensioners was noisier than the rest so that was the initial target.

I did a little research and there is a replacement tensioner and idler wheel set for the princely sum of £200. The set also included the Supercharger belt which I had just changed so I did not want to go down that route. I then came across a post on a Jaguar forum. - Bearings & Idler Wheel

The solution is to replace the bearings in the various wheels and idlers and the post included details suitable part numbers. I ordered four bearing for a total of just over £14 so a big saving. I changed one idler wheel bearing without removing the Fan assembly but you do have to remove it so you can get to the bolts holding some of the other idler wheels on. This is reasonably straight forward.

XKR Ratchet Spanner
Ratchet Spanner from Aldi

Once you have struggled to get the two nuts off of the fan assembly, they are not tight or anything just there is no access to them. I used a small ratchet type ring spanner that also could be angled enough to get into the tight spot. (from a cheap set I brought myself last time I was in Aldi)

I removed the top plate holding the radiator in position and I found I needed to remove the top radiator hose and the air filter top to allow the Fan Assembly to be juggled out of the way. Once the Fans were out of the way you can undo the various wheels as needed.

I used a socket, a long threaded bolt & nut  plus a large metal bar to pull out the bearings and tapped them back in using a suitably sized socket and hammer.

Made famous by Haynes, Reassembly is the reverse of removal, and once all back together, topped up the coolant that I lost disconnecting the pipe and went for a test drive. All seems well except for I started the engine with the plug into the air box unplugged so I have an engine MIL light illuminated. I’ll see if it goes off on its own after a few journeys otherwise I’ll plug in the ODB2 device I have and turn it off myself.

Back on the road and no tweeting …. for now anyway!

No good 🙁
A couple of days later, the noise was back see the video below:

BD – The Haystack Analogy

Find the Needle in the Haystack
Find the Needle in the Haystack

Finding the “Needle in the Haystack” is an often used analogy in discussions regarding Big Data and analytics. The problem is that having more data (A bigger Haystack) does not always mean you will find more needles. In fact it may make it harder to find those data nuggets you are searching for. I came across some great discussions/posts in the comments section of an article on “The Register” the other day. The story is “GCHQ mass spying will ‘cost lives in Britain,’ warns ex-NSA tech chief

Some of the analogies and references used in the comments included

  • Setting fire to the haystack and using a magnet to pick up the needles
    • One problem was that the heat may damage the needle (corrupt your data)
    • The terrorists could change to Carbon Fibre Needles so your magnet will not find them!
  • A far more distasteful analogy was that of a septic tank where you are looking for a chunk of chocolate but to find it you have to sample everything. Yuck!

In summary I guess the idea is to add as much “relevant” data as you can into your Data Lake but try not to pollute it with pointless additional data that makes it harder for you to find what you search for. The challenge is then, what is relevant data!

XJ6R – 2016 here we come :-)

2016 New Year by

Welcome to 2016, hope it treats you well and you keep healthy 🙂

I did spend a little time in the garage on New Years Day and did some work on the XJ6.

  • Cleaned up and sprayed a bit of clear lacquer onto the two washers and large nut heads that are used to hold on the bulkhead heat insulation panel. See the before and after pictures below
    • I also tidied up the wiring a little
  • Made up a gasket, found some suitable bolts and installed the brake pedal assembly and servo unit.
Washer "Before" image
Washer “Before” image
Right hand "shiney" Nut & Washer
Right hand “shiney” Nut & Washer











Next up – XKR

XJ6R – SuperCharger Conversion Decision

SuperCharger Requirements

Just swapping out the engine for the SuperCharged version of the AJ16 4.0 Litre engine has a lot of consequences. These then have knock on consequences that can make the swap very difficult. I decided to use the AJ16 engine and Automatic gearbox and the minimum amount of the XJR electronics needed. The alternative is running a standalone engine management system although I may have to go this route if I cant get away from the dreaded “limp home mode” with the original electronics. Very little else from the XJR is needed, I hope 🙂 Below is a list of the bigger issues and I will mention my thought process for each one and the proposed solutions.

  • The XJR6 engine requires different engine mounts
    • I tried with the standard XJ6 mount and the engine is too far forward so the oil pan hits the subframe
      • I purchased XJS facelift mounts (they have the AJ16 non supercharged engine) and hooray the engine will fit into the space. It it very close to the passenger inner wing because of all the intercooler add-on parts. It is also “swapped over” compared to the NA engine for inlet and exhaust. On the NA engine the inlet is the drivers side and the exhaust is on the passenger side in the engine bay. Now it is reversed.
      • Swapping the exhaust over should be OK as there is an offset Y shaped part than can just be rotated. Not sure on the whole exhaust setup yet but thats a minor issue to resolve anyway. I will use the AJ16 SC down pipes and Catalytic Convertors for sure.
    • I will use the XJR radiator, Fan Assembly and the majority of the XJR ancillaries as needed
    • Power steering pipe will need modification to fit the old steering rack. Need to research the pressures involved too.
    • I will use a mixture of old and new electrics as needed. For example, the horn, lights, indicators will be completely using the old XJ6 Electrics whereas because there are now electric fans, these will be driven via the newer XJR electrics. The Reverse lamps will use the same wiring as old but be driven by a new relay that is engaged via the reverse switch on the J-Gate lever assembly so a mixture of old and new.
    • By far the biggest challenge is the Engine electronics. I have researched this quite a bit even before I decided to purchase the XJR. I aim to use the following ECUs and wire them up as if the engine was still in the original car. I will mount the Control units in the passenger footwell area
      • Engine Control unit – obviously
      • Gearbox control unit – obviously
      • Security and Control Unit – to get over the inhibitor issues (will use the original key and reader but see more on that in later post)
      • Body Control unit – needed along with the others to run things like the starter and gearbox interlock process
      • For the gauges and warning lights I will use the circuit board that drives the XJR display but merge the wiring into the old display to keep it looking like a classic Series 2 XJ6
      • Getting the wiring into/out of the engine bay is challenging.
        • I will use the hole for the LHD steering option that currently has two vacuum pipes for the AC system running through a circular plate and make another hole for a single XJR bulkhead connector. Not all the connections are needed but enough to make it worth wile using one of the original connector pairs.
        • Need to add some fuse boxes into the engine bay. The XJR had 2, I think I can get away with 1 as I will not require all the extra items such as the lighting, fog lights etc that were in the engine bay area because those are already catered for on the XJ6 wiring loom.
    • Not yet decided what to do about the fuel vapor evaporation control system. The system on the XJR is quite complex with a carbon canister, a pump and a valve. This is combined with pressure sensors in the fuel tank as well. As I am using two tanks now and neither has the pressure sensor I will have to do something different and maybe send some fake signals to the ECU to stop any limp home issues.
  • The XJR gearbox requires different mounting points and the change to a J-Gate gear leaver assembly
    • Need to modify the mounting plate to fit body
    • Need to modify the central console and the underlying metal work to fit the J-Gate set up
    • Need to upholster/cover with veneer the modified console to hide the changes
    • Need to modify/merge the old/new wiring to manage the gearbox interlock and reversing lights
  • Drive train – I am initially going to use the old XJ6 rear end as it stands. If the gearing appears poor in use and I want to change it I can revert to an outboard braking system and the full cage from a facelift XJS. Maybe the 4.0L or even the V12 setup.
    • The propshaft will need modification. The XJ6 is a long wheel base and a cursory look shows that there is a bearing assembly half way along the propshaft with a UJ joint. This means that I could potentially use the front half of the propshaft from the XJR and the rear half from the XJ6. Later investigations proved that although this looked like a simple change, the sizes of the UJ’s are different. The XJ6 is imperial and the XJR is metric. Still it was worth a check 🙂 I will get the propshaft sent off to a specialist to modify it to fit. I need to get the engine and gearbox fitted in their final resting places so I can measure up properly.
  • Knock on effects
    • Fuel pumps – The XJR has two fuel pumps inside the tank with the second pump to boost the pressue over 3,500 RPM. I did look at the potential use of the XJR tank and did remove it from the car but I will stick with the original tanks and set up a new system using a high pressure external pump and the change over valve system.
    • Speedo and Mileometer –  These are mechanically driven by the old gearbox on the XJ6. On the XJR these are electronic with the speed signal pulse being generated by the ABS module and its sensors. I am not going down the ABS route so that will not work out for me. There is a output speed based square wave signal from the XJR automatic gearbox however it is 15 time faster than the signal provided by the ECU unit.
      • My plan is to use the small electronic gauges from the XJR inside the original gauges for speed and set the mileometer just under the dash somewhere discreetly to show the milage. I will use a mixture of the old and new sensors/gauges as required. The goal is to for the gauges to look the same as they do on a Series 2 XJ6 but working. I will wait to finalise this until the engine is in and working.
        • I did test fit the electronics to the original speedo and it looks like this will work out OK. The photos below show the initial attempt. It still requires some finessing so the plastic is not under stress, but it looks like I can get it to fit inside the original gauge.
        • Motor test fit
          Motor test fit

          Initial motor test install
          Initial motor test install
        • One issue is that there is a gap for the trip and mileometer in the gauge. I resolved this by taking a photo of the original gauge with the trip and mileometer in place. I then printed out the photo and stuck it behind the faceplate aligned with the holes. It looks quite effective even it it will not rotate. Note this test configuration used poor quality photo paper so it seems to have gone a little reddish whilst sitting in the garage for about a year. I will do another one and maybe laminate or spray clear coat over it to keep out the elements.
        • Speedo with fake mileometer
          Speedo with fake mileometer
        • I will need to divide the input signal down to reduce the rate. It is directly 15 times slower however I have yet to do the calculations regarding the rear end differential ratios and tyre diameters so it will need adjustment. To logically divide a signal by 16 is pretty straight forward so I will start there and adjust the analogue feed to the gauge to adjust to get a representative speed.
        • The circuit board that drives the XJR gauges is planned to be installed to the left hand side of the drivers foot well under the console

Wow thats a lot to contend with but thats the plan with no doubt loads more to decide upon and try to resolve as I go along.