Gramming (google programming)

I find it interesting how the IT world has changed over the last 15 years. 15 years ago if you read a chapter in a C programming book on how

to create a linked list or knew to add the word 'virtual' next to a C++ classes destructor - it was an attribute that someone would employ you

for. Now days with the growth of IDEs that check you syntax, static code checkers,  IDE autocomplete, Wikipedia, answers websites like

stackoverflow.com and search engines like google - programming has changed in some regards to cutting and pasting an error message/search

terms and finding the solution. 

So it seems to me its more about having the correct keywords to search for and knowing where to look.  I find this worrying because programming used to be a highly skilled craft like studing to be a carpenter or a goldsmith, in the dot com boom it was the

skill that everyone wanted, but how will the job change in the future.

Big data in finance

great video - http://www.infoq.com/presentations/finance-sql-nosql-newsql

All technologies mentioned where open source


Google releases papers on their technologies, open source world copies their implementations
ie. everyone else is playing "catchup"

GFS(2003)  -> HDFS
MapReduce(2004) -> Hadoop
BigTable(2006)  -> Hbase / Cassandra / any column family database

Stream processing frameworks


Percolator(2010) ->

• Twitter Storm - twitter has taken 3 years to create this since paper was released
• yahoo S4 - apache incubator project

spoke quite enthusiastically about "storm"

Realtime("response in seconds") sql-like query functionality

Dremel(2010) ->

• Cloudera impala  - presenter comments that this is "most mature" option
• Hortonworks Tez/Stinger  

Scalable graph computation framework

Pregel(2010) ->

• Apache Giraph 
• Titan 
• Microsoft research trinity

Search engine
business example: finding a clients CDS ISDA (can take months for ISDA to be located, client's gets tired of waiting for it and deal is lost)
business example: ability to search research document - pretty certain most banks use

• lucene / solr
• lily (on top of hbase)
• datastax (on top cassandra)

Future for hadoop is "reatime"
concept of using twitter storm + zookeeper  + hadoop to achieve a realtime CEP engine

CEP useful for "event driven investment banking use cases"

Lambda architecture (Nathan Marz)
mentioned upcoming book - http://manning.com/marz/

<insert diagram here>

batch layer -
use hadoop to preprocess data into QFDs - (question focused datasets)

speed layer - process deltas

both share same data store (HDFs) so data can be shared

CAP theorem
12 years later - creator of theorem is disputing its application by the creator of the theorem
http://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed

Four Vs
reference to Clouderas four Vs - "to help you sell big data in your firm"
last V - is "value"

Montecarlo simulation using Hadoop
"map" part of map-reduce can run a single simulation
if the nodes that are running the "map" part have a GPU - this could be used
or you can just use raw CPU power

you just need to implement the standard map reduce interface
mathermatica has a new release that abstracts aways GPU

Time series data 
this data tricky
kdb - can be hard to scale
open ts db - open source time series database
I think cassandra is good for timeseries - unlimited number of columns (Nicolas covered this in Acunu training)
presenter referenced concept of processing different scenarios including time series data on a graph database as a possible future idea

Clustering / classification
could classify stream of research documents and route to the correct trader
can use mahout - extract sentiment

Concept of data hub
all data can be stored in one place
eg. correlation of weather, commodity prices - so that many different teams have access to this data

its not difficult to get 1000x3TBs drives and have a petabyte of space
you can run 32 core cluster at home

provide R for statistics

Hortonworks porting hadoop to Microsoft technologies... - eeuw

video on how to implement a ESB in finance

http://www.infoq.com/presentations/Large-Scale-Integration-in-Financial-Services

Great video with ideas how to improve ESB (enterprise service bus)

• fix - is used for front office
• fpML - middle office 
• SWIFT - interbank transfer (done at end of day)

concepts:

• store message in its raw format as "clob"
• could be transformed into pojo
• create a xpath like language that can refer
• xpath 2.0 can also check message type
• message routing etc can be based on xpath
• provided the message format and version is know - we can always go though the messages and extract additional fields. ie. make those fields available as an "index" fields 

ESB could just be a in-memory cache - (which may persist the messages)

Bench time as part of life working for a consultancy

I am starting to get used to effects of bench time

• the first week on the bench is great - it gives you time to recouperate from the project (especially the final sprint to get the project completed)
• there is a mental taxing part - the stigma of not earning fees and the uncertainty of whether a project you have been forward for will come actually happen 

For a consultancy that has both permanent and contract staff - The deck is stacked against you at review time - consultancies measure part of your performance as 'time billed'. However if you are permanent, I believe they tend to place your name against projects that are more volatile in terms of whether the project will happen or not.

Also after enough time on bench, your standards tend to be worn down. Eventually you will take any project you are offered - just to get off the bench. Projects that you would have never considered in your wildest dreams - now start to look attractive.

This is another side affect - you might hope to work on projects of type Z and your name has been put forward for a Z projectYou sit on the bench for a month, clocking up negative billed time (possibly passing the limit on your review success criteria), eventually you just end up working on whatever you can.  

The secret is to use your bench time effectively - this can be difficult if you dont know what project you will be on next week. Which asset class should you read up about, what technology should you study - if you don't know which project it will be. You need to have a plan of skills to learn and personal projects focus on getting these done.

cottage industry of providing feeds

In the banking world there is a  IT ‘cottage industry’ of providing / consuming data

 

Nowdays you need ‘director level’ approval to give another team access to your database – this is because giving someone a feed from your system is like entering into a lifetime agreement – you team will be blamed and have incident ticket raised against them each time this feed is not present or the data is believed to be incorrect. When a team requests access to your database, simply push back by saying “there is the director responsible for this system – please obtain approval from them”.

My life as a gardener

Life working in bank is a little like being a professional gardener.

Each ITPM(IT project manager) in the bank is given a small set of systems that they are responsible for. This is smallish and rectangular similar to a back garden of a terrace house. Colleagues that sit next to them are similar to next door neighbors.

Rules for lawn owners

• Do not peer over the fence and start commenting on someone elses back garden and state of their lawn.
• Do not start mowing someone elses lawn - you will either end up becomming responsible for it forever or upset the owner.
• Be careful touching anything that represents an interface between your system and the neighboring system - breaking an api or feed - will result in your being blamed for an outage
• Focus on creating a clean lawn with nice straight stripes - wax on, wax off

Rules for gardeners

• Do not touch the lawn mover when it is being used - wait till after hours before touching anything. If you do not, you risk loosing a finger.
• If there is a lot of work to be done, schedule it over a weekend
• Be careful of anything that is offered to you - its probably a either a rosebush or a set of clippers - you may end up loosing a finger
• As a professional gardener you may work across different owners lawns - in order to implement a drainage system - but be careful working across streets as these are controlled by different managers

Standard second question - Whats the ETA?

In the City there is always a standard "second question".  For an incident or a feature - the second question is always "Whats the ETA?"
The next question might be "Is it a tactical or strategic solution?", possibly followed by the "Whats the escalation path for this?" or "What else are you working on?"