Bar opens at 8

KiLlEr · Post by **KiLlEr** » Fri Mar 28, 2014 4:00 am

@HF: The logs are going to be loaded into a graph database (cube) and the output sliced and diced. We're storing lots of contextual information with the log messages as well as runtime statistics.

Basically, an error can be traced back to the exact period of time and cross reference the performance of the machine at the time of the error, the offending process, the inputs to the offending process, messgage queue depths, database load, etc etc.
Then compare it to the same time period on other days, see if there was any configuration or deployment
Changes that occured prior to the failure, etc etc...

Scary powerful. And insanely fast.

@Tau: yeah I was thinking of licensing Inner Calm to a pron publisher. LOL

Post by **Taurec** » Sun Mar 30, 2014 1:28 pm

What counters are you using ? .. (wmi, snmp or custom magical sensors?)

KiLlEr · Post by **KiLlEr** » Sun Mar 30, 2014 1:40 pm

NLog and log4J based custom agents with customized loggers to write the logs directly to the graph database. We read from JMX, WMI or the Api's exposed oracle etc.

We read a few hundred counters except for WCF. Our application logging framework has a timing block api which allows use to track individual transactions from start to finish accross distributed processes. Each log entry carries with it a timestamp, process information, thread information, and the parent block so it can be traced. System events are logged similarly. Database load, queue depth, network utilization on a nic, are polled at intervals and time stamped and logged with context information.

All this is dumped directly into a graph database. Data is migrated off after a period of time and performance data is aggregated into time slice. After a period of time, the time slices are aggregated into larger time slices depending on age. Really old aggregations get pulled off and archived.

The amount of data is frightening.lol

Post by **Taurec** » Mon Mar 31, 2014 7:32 pm

Sounds awesome.
What BI tools are you using to make sense of it all?

HELLFIRE · Post by **HELLFIRE** » Tue Apr 01, 2014 6:13 am

@Killer
My head is spinning already.... just stop right there.

Well, today was the last official day of support for the customer... so me going into the office tomorrow,
well if the phone rings, I'm not picking up. Wish the new support guys luck, but if they're not going to
ask questions, that's their business.

Now have to get myself aquainted with the new role I'm in... I have to admit, alot of fear and "deer in
headlights"... we'll see how it goes.

Tall and Cold...

KiLlEr · Post by **KiLlEr** » Tue Apr 01, 2014 2:58 pm

Sorry Hellfire.

We use a custom gui because its tied into our monolithic software., but we are looking at some available web-based ones. The one from Oracle is being considered, since we have a relationship with them.

Post by **Taurec** » Wed Apr 02, 2014 7:56 pm

Check out dynamicai we get pretty good results from that BI tool..

Hellfire, have fun just see it as a new challenge .. so what is written on your new business card now?

HELLFIRE · Post by **HELLFIRE** » Sun Apr 06, 2014 5:01 pm

Just have to post this up for posterity... into the last couple hours before our support
ended and it moved to the new group, the question was posed, by one of their staff
members apparently, "What's a WAAS ?" Wasn't in the office or on the call, but apparently
the customer was beyond livid by this point, and I SO would've wanted to answer
"Well, I WAS supporting this, but I'm out of here..." and hung up the phone.

Last couple days have been spent applying for IDs and dealing with the silence in the
office. Friday afternoon was deboarding another person... the teamlead of all ppl, and
the whole reason our support group existed in the first place... there aren't the words
to describe what was going thru my head watching him clean his stuff up.

@Taurec
Nothing actually... even if I had a business card. I'll still call myself what I've always
called myself, "Network Analyst, <compay name here>."

HELLFIRE · Post by **HELLFIRE** » Sun Apr 06, 2014 5:08 pm

Just as a crazy thought, Killer, would the thing you're working on help out on this scenario?

The issue that we're facing is there are error in the application where a thread is stuck at the socket read level which potentially may cause a database table lock.

Here's the error trace from the application:

[5/29/13 13:06:57:588 EDT] 0000002c ThreadMonitor W WSVR0605W: Thread "WebCont
ainer : 11" (000006bb) has been active for 703191 milliseconds and may be hung.
There is/are 2 thread(s) in total in the server that may be hung.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:79)

Is there any way you'd be able to trace that socket read during that time (long waiting socket read or drop packet etc. just to point a few). Appreciate your help

Don't bother trying to read too much into what the person was asking for... this was one of
those "Wall Of Shame" gems I've kept over the years... but I'm wondering if what you're working
on is in the direction of what this guy was asking for.

Tall and Cold... not quite a lazy weekend, and I have to make a run now for "IT support for family" *sighs*

KiLlEr · Post by **KiLlEr** » Sun Apr 06, 2014 7:53 pm

@hf: not only that, I would be able to track that back to the transaction that was involved, the person who submitted the work, and every step the transaction took. Also, I can cross reference it with the load on the server, what was running at that instance in time, how much resources were being used for each process, the queues the trans action stepped through. etc. etc. etc. Database load, tables involved, EVERYTHING. It's scary, especially the rate at which we suck up storage space (*_*). LOL

Basically, we can (ultimately) trace a fault at both the software and hardware level, as well as compare the time slice against historical data to see if the fault was caused by a "growth" over time of resource utilization, and reference configuration of deployment changes that may have played into it.

HELLFIRE · Post by **HELLFIRE** » Mon Apr 07, 2014 4:38 am

Do me a favor then Killer, write up a price quote, I'll find the email trail this was from and send it to them.
Pad the quote as needed... all I ask is a 20% share, fair enough?

Had to clear off some unwanted adware off one system, then rebuilt a "MBR is compressed" problem on a 2nd system
that I've been procrastinating on since October of last year. THEN I got the bright idea to "update" the 2nd
system... DON'T ask me how a simple Adobe Acrobat 9 -> 11 update took ~2hrs to complete, AV definitions took ~3hrs,
and AFTER I finally got out of there, the system was still pulling down updates *sighs*

Post by **Taurec** » Mon Apr 07, 2014 6:58 pm

MBR Compressed ... that's kinda in the same category as removing the system rights from your bootdrive

At least he tried to explain...
"But I thought removing all entries from security would harden the system"

And it did .... if nobody can touch it .. a virus/exploit wont lol

Joe, how do you copel with the latency reading from all those sensors?

HELLFIRE · Post by **HELLFIRE** » Sat Apr 12, 2014 3:47 am

...spent the week getting IDs for the new stuff I've been doing, and doing ALOT of reading.
From what I gather, what this offering is basically for customers that don't have the time, money and/or expertise to build and maintain their own datacenters, so they get (rack)space in one to build out their infrastructure.

Think a sandwich, with the meats, etc. as the "shared" stuff, and top half is for the vendor to come in and manage / maintain this, and the bottom half is where the customer can connect in.

If your head's spinning, guess how mine feels after this last week...

Tall and Cold, helping a friend move tomorrow.

Post by **Taurec** » Sat Apr 12, 2014 9:36 am

Sounds awesome .. I had to deal with this on my own years ago.
I think it was around 2004, all I got were the power plugs and one ethernet cable to start with.

My tip would be ... virtualize everything !!

HELLFIRE · Post by **HELLFIRE** » Wed Apr 16, 2014 12:31 am

Moving's done, but like any other move, ALOT of injuries, insanity, and frayed tempers by the end... it got
all done though.

Other than that, not much new. Doing alot of reading ... and occasionally sneaking into the old inbox to
see what kind of correspondence we get but don't have to do since we no longer support said client. I
STILL laugh at groups that still slap us on an email trail and expect service -- apparently the OOO warning
"this email no longer supports Account X, please contact Y" isn't registering with everyone right.

Oh yeah, next time Noir drops by, wish him a Happy Eviction From the Womb Day, as Apr 13 was his birthday!
Extra Drinks from the house pls!