Logback hack to log different levels in the same package to different appenders

Given the long title you could think that this is going to be a blog about a ridiculous edge case. It isn't, honestly! It is a blog about an edge case, but not ridiculous and not as far fetched as you think. Let's say that you have a class logging on DEBUG and TRACE. The debug messages contain (frequent) checks, but lack the full information needed to analyze the problem. This information (say a full xml message) is logged on TRACE level. Now your friendly system administrator decides that he needs this trace logging to go into a different file from the debug logging. He has good arguments and you don't want to pick a fight. Logback provides a lot of configuration options that you can do lots of neat things with. But it can't fill this requirement. Really? Well, there is a hack.

Found in the wild: uncertainty

Let met just keep this short and let the code speak for itself:

1 2 3 4 5 6 7 8

// The next fields counts the number of time the decide method is called   //   // IMPORTANT: This field can be updated by multiple threads. It follows that   // its values may *not* be incremented sequentially. However, we don't care   // about the actual value of the field except that from time to time the   // expression (invocationCounter++ & 0xF) == 0xF) should be true.   private long invocationCounter = 0;

Three letters apply here.

Posted via email from Iwein's braindumps

Confession of a developer junkie

I like splitting wood. It’s a simple physical activity and I’m good at it. I can tell you exactly why I like splitting wood; it is very similar to why I like doing some other things, like building software for example. There has been a lot of discussion in software development about this thing some people call craftsmanship

When you split a log (assuming you have a decent tool) things are quite simple. There are a lot of physics involved, like the direction of the grains, the cohesion between them. The moisture is important too, the momentum of the axe, the angle… But when I swing I don’t think of those. I just look at the log, know where I want to hit it and then it happens the way I saw it in my head. It doesn’t always go right, but it does a lot of times. When it goes right I know it before I register it. Right at the moment the axe has its maximum momentum and the direction is set I know there is nothing to correct and the scenario can play out the way it should. And then there is the moment when the axe hits the wood and sinks in effortlessly. There is the well articulated “chk” immediately followed by a “pang” as the tension makes the log snap apart. If you hit as hard as you can you will send two parts flying away, but that doesn’t give me most satisfaction. I like it when I hit just hard enough and two parts end up standing next to each other. I can immediately go on and take another swing without stooping over and straightening the logs. That feeling is the most addictive thing I’ve found in life. I can continue splitting wood far beyond comfort, just to get another fix of that feeling. I am not alone in this I believe.

A long time friend of mine is a baseball player. He told me a similar story about hitting a ball just right. The pitcher throws, not too slow, but you catch the ball with your eye. You swing at it and as the bat cuts through the air, you know it’s going to connect right. Then the ball hits the bat so that you hardly feel it. You just year the “ping” and you know it’s a home run. I’m not a baseball player, but apparently there is nothing like that feeling. I believe it is the same feeling that I get from splitting logs.

A related thing has been described before by Csíkszentmihályi Mihály. It is referred to as “Flow” or “The Zone”. If you haven’t seen Csíkszentmihályi’s work you definitely should have a look (feel free to copy paste now to avoid spelling errors). I believe this immediate feeling of joy as you do something right lies at the basis of flow.

With some effort I can get this feeling when developing software too. It is even possible to share this feeling with a whole team of developers. When that happens I love my job so much I’d do it for free until I starve. With some more effort it is possible to do this on a project that actually delivers business value to a customer. This means I can do this great thing without starving if I play my cards right. I need a way to get the people that stop me from doing this to get a clue and let me do my thing. You probably know the types I’m talking about: it’s the manager up high that makes me work on a virtual desktop that runs vista; it’s the architect that forces me to use a five year old version of a bad framework and makes me deploy to Weblogic 5; it’s the customer that waves three years old specs at me when I ask him a question about how he wants you to implement a feature. It is also the colleague that refuses to talk to me about my real problem if it is not related to cool new obscure technology X that somebody blogged about this morning.

If I can get these people to grow up by talking to them about Agile, XP, or Craftsmanship, or Lean, or Oompa Loompa’s; I will do it. I’m a junky remember? I want to get my daily flow fix! When Robert Martin said we were tired of writing crap, it didn’t ring true with me. I’m not tired of it at all. I’m prepared to write a lot of crap to get it right just once. If we do have a nice bit of marketing on the word craftsmanship I’m fine to surf that wave for now. Next year we’ll call it Real Engineering, the year after Real Development. Why the hell not. As long as some of us get what it is about and they are in my team I’m totally cool with it.

Posted via email from Iwein's braindumps

Ja tegen proteïnen, nee tegen vieze shakes

Ongeveer een half jaar geleden kreeg ik last van mijn rug. Het advies van de dokter en fysiotherapeut was om zo veel mogelijk te bewegen en zo weinig mogelijk achter de computer te zitten. Omdat ik vastbesloten was, mijn rug (en buik) zo snel mogelijk in vorm te krijgen, heb ik het advies opgevolgd. Deze blog heeft daar enigszins onder geleden, maar omdat ik nu een slanke en fitte dertiger ben, mag ik weer lekker lang achter mijn computer zitten ;-). 

Gezond eten en regelmatig bewegen zijn onmisbaar om een gezond lijf te krijgen en te houden. Als je eenmaal lekker aan het bewegen bent geslagen, wordt het een sport om steeds iets meer te tillen en langer door te gaan. Op het feestje van Wouter, kwam The four hour body' ter sprake. Om een goddelijk lijf te krijgen hoef je volgens Timothy Ferriss slechts 4 keer in een maand te trainen en moet je 100 gram extra eiwitten per dag eten. Als je bedenkt dat er in een eitje hooguit 7 gram proteïne zit, moet je dus al gauw 14 eieren per dag eten... Ze zijn niet gemakkelijk weg te werken, want eieren vullen goed. Je  gaat van eieren enorm stinken. Bovendien is een eenzijdig dieet niet gezond. Dus op welke andere manier krijg je dan 100 gram proteïnen per dag extra binnen?

Curing distributed Agile Illnesses: Diagnosis

In my session on 'When Agile Offshoring fails' at AgileNCR this Saturday I decided to harvest ideas from the attendees on different illnesses. The results were beyond my expectations; in this blog I will share a few of them and give a few preliminary comments on the result. The general plan is described better with the following diagram.

First I would ask the attendees to come up and write down all the symptoms related to failing distributed agile teams on a sticky. These we put on one part of the walls of the room. Then I would ask them to write causes for symptoms on stickies and put them on another part of the wall. Finally we would all try to come up with a possible mitigation for each cause. It worked beautifully, let me show you.

Easy peasy websockets with node.js and socket.io

With our Web Sockets incubator we had a rough start on Atmosphere grinding through the samples. There is no intention to bash Atmosphere, it looks like an interesting approach. However, it should be a lesson to all aspiring framework builders out there: getting your samples in top shape matters a lot. This week we tried out socket.io and I wasn't too hopeful as I had planned all kinds of things throughout the evening. The result of a 40 minute pairing session with Sonny Gill was pretty good and I'll share the basic highlights here in a very short post.

First we installed node.js and used npm to install socket.io. Next we moved on to create a simple app that exposes a socket on the server. Finally we created a client that opens a socket and sends some test string over the wire.

The code you can find on github as usual: https://github.com/xebia/app-incubator-web-sockets/blob/master/socketio-sample. All in all it is less than 50 lines of javascript and html and it simply works as expected. Not bad for less than an hour of fiddling! In our next session we'll look into the possibilities of streaming video over Web Sockets, or maybe we will look at another framework to see if it performs any more interesting magic tricks. That's all for now, stay tuned!

Posted via email from Xebia App Incubator

Force breakthrough by forcing mistakes

A week or so ago I started to play Math Workout to rekindle my degenerated calculation skills. After a while I got stuck at a certain speed, making no mistakes anymore. I tried to push myself to go faster, but without making mistakes but that just didn't work. Then I tried a different approach: I forced myself to go fast, mistakes or no. After a couple of sessions, I went back to making no mistakes and to my surprise I was around 8% faster. I have a theory about why this is and I'm applying it to life without hesitation from now on, so read on if you want to share the fun.

<!-- more -->

What's Math Workout

I'll give you some background on Math Workout first, so you can understand the example. Math Workout is a game that let's you solve simple calculations and times you while you do so. The main goal is to be as fast as you can. For example, on easy level the hardest addition is 8+7 and the hardest multiplication is 4*6 (I'm considering 5*6 easier). Dead easy as you can see. The trick though is that you need to solve 20 of these problems in less than 23 seconds, and I can tell you that's tricky enough because you need to read the problem, come up with the answer and type it correctly in about a second.

What goes wrong if you don't make mistakes

Somehow I got into a rhythm. See the numbers, see the operator, answer pops up, type it. There are probably a few more steps that I was doing unconsciously, but the point is that I was taking a fixed amount of time for each step. Sometimes I found myself breaking the rhythm just before I punched the wrong number and then taking some fixed amount of time again to correct myself. Whenever that didn't happen my time would be between 26 and 27 seconds consistently. Basically I had put a few safety checks in (which I probably didn't need most of the time) that were slowing me down. Whenever I pushed myself, I would occasionally parse the wrong operator, type the wrong number, pop up the wrong answer and quickly revert to the strategy that worked. Once I noticed this, I knew I needed to break the pattern.

Forcing errors

I decided to change the pattern, and ignore the mistakes at first. I decided to ignore the operator check and assume that my brain would be able to pick up the numbers and the operator in one go. I made 5 errors out of 20 questions and decided that I was bad at picking the operator, but it must be working somehow, otherwise I would have seen more in the range of 10 mistakes. I gave myself positive feedback by thinking I was a smart cookie for figuring that out.  I repeated it a few times and saw the errors go down and the times with it, more positive feedback. After a mere 5 runs I focussed on making no mistakes again, and I was below 25 seconds. Mind you, before that, trying for over 50 runs there was no way in hell that I was going to go under 26, my top times were averaging around 27.5 seconds. By forcing the errors I had broken the pattern and found a more optimum way of behaving. Much faster than waiting for some luck would have done.

What's going on here

When you make a mistake you tend to give yourself negative feedback "aww shoot!" and this tells you to avoid the behavior that led to this the next time. When you don't make mistakes and you have a better time than before you give your self positive feedback and reinforce the behavior that led to the result. When you try something new and it doesn't pay off immediately, you will give yourself negative feedback. We crave positive feedback, so we keep on aiming to get that result improving, even if our method is flawed and we're in the wrong rhythm. It's hard to break a pattern that reasonable results but is suboptimal. The main reason here is that when left to it's own self learning devices, the variation in methods goes down rapidly. It takes a conscious effort or a lucky coincidence to break out of the suboptimum. The lucky coincidence becomes less and less likely the more you reinforce the suboptimal behavior. This is caused by a thing called Extinction Burst which is intended to pulls you back into the behavior that worked in the past.

Download now or preview on posterous

PastedGraphic-3.pdf (21 KB)

What's to learn?

It pays off to try things, even if the first results are not impressive follow through for a bit and see if you might just be crossing a chasm and things are better on the other side. I've seen this pattern not only in my own behavior, but also in the behavior of groups of students and teams of software developers. As long as we didn't plot the graph of all possibilities the only way to find out that behavior is suboptimal is to explore and find a better type of behavior. With Math Workout this is easy, because if I can't make the required time my behavior must be the problem. If there is no required time and nobody knows which goals to set (like in Software development, or any kind of creative work really) you can never be sure of what works best until you've literally tried everything. The trick is to find a way to reward yourself for trying the new thing that doesn't relate too much to the old reward of squeezing every last drop out of a poor method.

I'm not saying you should be trying everything right now, but how about trying one thing each week? That would already be a huge improvement for most of us I'm sure.

Posted via email from Iwein's braindumps

Olijfolie

Het is in Nederland niet makkelijk om goede olijfolie te vinden. Als je de spullen van de supermarkt gewend bent, weet dan dat er de gelegenheid is om een ster te winnen simpelweg door het gebruik van betere olie. In deze blog ben ik op zoek naar een goede, betaalbare olie en waarom de beste olie die ik kon vinden niet in de winkel te koop is.

Bij het produceren van olijfolie worden verschillende extractie methoden gebruikt. De belangrijkste factoren zijn de lage temperatuur (eerste koude persing) en de manier van malen waarbij zo min mogelijk de pit en schil van de olijf meegeperst moeten worden.

Deze beide criteria zijn in directe tegenspraak met de kwantitatieve grenzen die het winstgevend produceren van olie in hun greep houden. Immers: hoe meer olie per olijf, hoe goedkoper de productie. Hierom worden latere persingen uitgevoerd waarbij de gemalen olijven fijner worden gemalen en worden gekookt.

Op zoek naar goede olie zijn veel speciaalzaken te vinden die wel prima olie hebben, maar deze olie is duur. 20 euro per liter is de prijs die de olijfoliespecialist in Culemborg bedingt. Ik betaal graag voor goed kwaliteit, maar als het wat minder kan (qua prijs dan) ben ik ook te vinden.

Vriend Tiziano nam mij een halve liter olie van zijn ouderlijk bedrijf in Italie mee. Vriend Guido bracht mij een kruikje olie van het vat dat hij in Kroatie had gekocht. De olie uit Kroatie kan zich gemakkelijk meten aan de producten van Kastelli, en de olie van Tiziano is beduidend beter (smaken verschillen, maar het zegt toch iets).

Even rekenen hoor… Er is olie die in het land van herkomst voor 5 euro per liter (of minder) verkocht kan worden, en beter is dan de olie die je voor drie keer dat bedrag in Nederland koopt. Deze olie komt niet op de markt (in het Italiaanse geval), en al helemaal niet op de internationale markt (in beide gevallen). Voor olie van de kwaliteit waar we het over hebben betaal je in de VS rustig 50 dollar per liter, en in het noorden van Europa is 40 euro per liter ook niet vreemd.

Het aanbod in het land van herkomst (Italie) is groot, maar de prijzen zijn daar zo laag dat de boeren de moeite niet nemen hun olie ter verkoop aan te bieden. De vraag in het noord-westen is groter dan het aanbod aldaar (anders zouden de prijzen niet zo hoog liggen). Het grote gat tussen de prijzen die de boeren in Italië krijgen en de prijs die in de winkels in Nederland betaald word is moeilijk te verklaren.

In elk geval, ik laat de conspiracy theories aan de lezer over, maar als je op vakantie gaat naar een warm land, vergeet dan niet een beetje olie bij een boer te kopen. Wel eerst even proeven hè?

Posted via email from Koken en Eten

Reaction to "Why ORCL was right to sue ..."

Posted via email from iweinfuld's posterous

Forging intelligent Teams instead of staffing Projects with intelligent people

In my experience with consulting and development, project teams (both Agile and traditional) invariably face issues with availability of team members. In one (rather long and messy) talk on craftsmanship Robert Martin came up with an interesting idea that I'd like to explore a bit more. It might actually solve this problem if used in the right way in a large organization. The idea is simple: organize into long term teams of upto 12 people and accept work from different projects as a team.

This is actually the way small development companies have worked for ages, and they have done quite well on it. The problem is that when projects and clients get bigger they tend to turn things upside down and instead of pushing work to teams that can handle it, they start pulling resources (eww I hate that word when applied to human beings) into the project. This process is called staffing.

There is a problem with staffing. People are not resources. The difference between people and resources (think coal, plated steel, money) is that they change based on changes in their environment. This ability is defined by Stephen Hawking as intelligence, and I like that definition. The big upside of intelligence is that it can actually solve problems for you and that is something that typical resources cannot do. If intelligence is the stuff that allows you to come up with solutions, resources are the stuff that you consume to implement them. A profound difference here is that coming up with a solution actually increases the amount of intelligence, whereas implementing a solution always decreases the amount of resources.

If managers pretend that people are resources and projects are black boxes that turn resources into implemented solutions they will get two things: shortage of intelligence and shortcomings in implemented solutions. The shortage of intelligence is caused by the fact that a project starts being about finding a solution and invariably more and more becomes about implementing it. The former makes you smarter (increasing intelligence) the latter dumbs you down (decreasing intelligence). If you only implement solutions you lose your ability to adapt to change, but if all you ever do is adapt to change you might go crazy. There is an optimum between the amount of change a person has to handle and the amount of time he needs to regain his footing, and this is different per person.

This is all well and fine, but how will we get those essential team members available then? I'm getting to it, don't worry.

First we put a team together thinking about a certain context, not a project. Then we let them establish a healthy environment for them to adapt to and increase their intelligence. This causes team members to be available to each other by default. The we let the project managers sell the work to a team, possibly multiple teams. Each team takes on what they are comfortable with but they stick together. Now this chunk of work might be too little or they might get bored with it. Then they take on some more work, possibly from another project, but they keep together as a team.

If a team member wants to leave this is a career move, not a ephemeral thing you can do for a couple of hours on friday if you feel like it. You can't be part time on one team and part time on another. If you want to take on some work you have to convince your team, or make your career move and be gone totally. Now the question of when person X is available to person Y is extremely simple. When they're in the same team they can count on each other. If they are not on the same team they might have a meeting some time. 

This should up a lot of things, I'm looking forward to try it. If anyone already has experience with this approach on a large scale or moving a large development organization into this mode I would love to hear some war stories.

Posted via email from iweinfuld's posterous

The Copy, then Rename pattern for file based integration

This is a blog for those unlucky many of you that, like me, had to solve the file integration problem. Two processes are integrating over a directory, one writes (W), one reads (R). Now the reader picks up a file while the writer is still writing to it. BOOM, R caughs and dies. Sounds familiar? Then read on.

I know I promised a blog on a deadline free utopia, but while I'm chewing over that one I need to jot down this thought. That's what blogs are for right to capture the excess of an overflowing mind?

So back to R and W that can't seem to get their processes synced. How do you prevent R reading that file that W is still writing? Locking will work, but it's clunky and tricky. Looking at the timestamp might work in most of the cases, but it is not guaranteed to work under high load. Can you say heisenbug? So, what can we do.

Well in fact the solution is too simple to spend more than a few sentences on. It is old, battered and scarred, and it will be around for a while in the future. Just Copy the file to a temporary name and then Rename it to its final destination. Maybe I'll include an image later, but you should get the point, right?

Posted via email from iweinfuld's posterous

BBQ experiment: Grilled pineapple

I'm having a barbecue with some Xebia colleagues next Tuesday. When I have a couple of guests over for the first time, I always like to play it safe a little bit, but the following two experiments just seem too easy, so I'll take the risk:

Grilled Pineapple (adapted from Ananas al Forno):

For the original recipe look at this old blog. I don't see a reason why it wouldn't work on the barbecue, and potentially improve the flavor given the right type of smoke. I'm thinking sage should work. 

Marinaded sweet potato

I have no clue yet how I'm going to pull it off, but I've made marinaded carrot shaslic's before and that worked out pretty well, so I'm guessing that it's "just" a matter of getting them done just right and finding the proper marinade for them.

I'll update this post with the story of success or as it may happen disaster after Thursday, if you have any tips, warnings or other first hand accounts please help me out before I make a fool of myself.

Posted via email from Koken en Eten

Please stop moving deadlines

I've been struggling with deadlines for a long time, and I've finally discovered the pattern that makes my struggle most painful. They move all the time. A moving target is harder to hit than a stationary target, so my theory is that the more deadlines move the more futile planning becomes. The good thing about time is that it (usually) only moves in one direction, but man has invented budgets to compensate for this glaring lack of complexity.

So what can we do? Well just never, ever move a deadline. I will go into detail some later post on how to avoid deadlines altogether, but for now let's just consider what happens if you make the deadline stationary.

When you refuse to move the deadline the thing will not be DONE.

This is not a problem. Nothing is ever done. Done is an invention by people who like tickboxes. When the deadline comes you just do what you planned to do at that deadline, or you conclude that you have failed. Failure is good. Failure is the only true done. If you have failed you are free to go do something else, probably something that you have found the inspiration to do when you were dragging around Doomed To Fail project X. A deadline that can't be moved helps you to fail early and this will save time and money.

When you refuse to move the deadline the thing will not be as good as it can be.

This is a misleading statement. If you're building a cruise control driver, "as good as it can be" is a well defined concept, but most of us aren't building those kind of applications. What we're building is usually the implementation of some idea somebody dreamed up with very limited input. It might be a very good idea, but it's not well defined. In fact it's our job to define it based on small hints the product owner is giving us. This makes "as good as it can be" a moving target. The longer we wait to deliver, the further ahead the dream of the business will get ahead of our actual implementation. And the bigger the disappointment when we finally do deliver what we have.

But just look at what we have now! We can't deliver like this, right?

Maybe you're right - you're probably not though. Delivering is easy and delivering something that sorta works is much better than delivering nothing at all. You don't know if the business can break even with the thing you consider crap before someone tries. It is your obligation to try this as soon as possible. Cheating your customer into thinking he's going to get rich is not professional.

Much more important though: the longer you wait, the more you're going to think you're not ready. Dreams go faster than reality and by definition you will never catch up. By delivering whatever you have on the deadline and not moving it you're delivering reality to the dreamers. The earlier you do this the more they will keep liking you.

Ok, no more moving deadlines. But what happens after the deadline?

Well, its simple. After the deadline people will invariably think it can be done better. If you deliver your ideas of how to do things better with the unfinished product, you might get a chance to fix the things you wanted to fix before delivering. The good news is that you will get that chance based on realistic expectations. And if you don't get that chance you're free to do something else, which might not be a bad thing at all.

Posted via email from iweinfuld's posterous

Stir fried spicy spinach

There's nothing original about stir fried spinach with a dash of other flavors, but this combination was pleasant enough to mention anyway. 

Per person:

• 200g spinach
• 2 cloves garlic
• 2 thai chilies (a normal chili pepper should work fine too)
• juice of half a lemon
• 1 tbsp grenadine concentrate (Ayurveda)
• arachide oil

Heat the oil over medium heat, add the ginger  in fine slices so it can get acquainted with the oil. When the ginger is getting soft add the garlic (bashed and chopped) and chilly, also finely sliced. After a minute or so, add the spinach and increase the heat. Stir until the spinach is done and add the grenadine and stir again. Finally turn of the heat and add the lemon. The taste is quite funky, but definitely worth the 10m that it takes to make.

Posted via email from Koken en Eten

Untitled

I made this for lunch yesterday and it was even better than I expected:

• 3 eggs
• handful of coriander leaves
• two hands full of mushrooms
• juice of half a lime
• 2 tbsp of skimmed milk
• 1 tbsp sesame oil
• 1 tbsp sweet soya sauce
• 1 tsp thai fish sauce
• rice oil (to fry in)

Chop the coriander and whisk all ingredients together except for the rice oil and the mushrooms. Slice mushrooms. Heat up a frying pan and toss in the oil and mushrooms. Fry until the mushrooms have lost most of their liquids and give off their aroma.Reduce the heat (or take the pan off the heat for the moment). Whisk the mushrooms into the omelet mixture. Make sure the pan is clean (if you have a good pan you can just stare at it for 0.5 seconds), add some more oil if needed and gently poor the omelet mixture into the pan so that the mushrooms are spread well. Gently fry the omelet on medium heat, turn it if you feel like it. When it's golden brown and completely solid you can eat.

(all the amounts in this recipe are wild guesses as usual, I don't believe in careful measurements when cooking)

Posted via web from Koken en Eten

Note to self: how to sparsely check out a git repository

Today I discussed with Dave Syer about some improvements in Spring Integration, which I had sandboxed in my messy github repository. A sparse checkout would be convenient, but we both didn't know how to do it. I even thought git didn't support it, but boy was I wrong!

iwein:si-sandbox$ git init

Initialized empty Git repository in /tmp/si-sandbox/.git/

(master) iwein:si-sandbox$ git config core.sparsecheckout true

(master) iwein:si-sandbox$ echo message-store/ >> .git/info/sparse-checkout

(master) iwein:si-sandbox$ git remote add origin git://github.com/iwein/Spring-Integration-Sandbox.git

(master) iwein:si-sandbox$ git pull origin master

The important thing to remember is that this also works with git-clone. Your index will still have everything, just your checkout (working copy generated from the index) will not. This is a big difference with svn sparse directories, and as usual it's an improvement.

Posted via email from iweinfuld's posterous

Using negative feedback to prevent disaster in Software Development

Invariably we are taught that when working with people we should give them plenty of positive feedback whenever they do something right and be very careful with negative feedback. I've never completely bought into this hugging hippie attitude, but it seems there is a good point there: if we're all nice to each other and keep a positive spirit, we get more done right? The problem with this approach is that it doesn't take into account that it might be important to not get something done. Some ideas are just terrible and they should be killed off as soon as possible.

When I had the idea for this blog I thought it would fit on twitter. "Positive feedback induces brilliance, negative feedback prevents disaster. The latter is sadly undervalued." i twote. It seems this could use a few backing arguments and today I've made time to provide them. I will explain why negative feedback is important and how you can apply it to your advantage.

What's the problem with positive feedback?
Positive feedback induces brilliance. Like the brilliant glow of a nuclear meltdown is caused by positive feedback. If you're doing something absolutely great, positive feedback will cause you to do more of it and it will cause others to try and emulate your behavior. Brilliant right? But what if you're doing something that is ever so slightly off target? Will positive feedback make you augment your course? Will it make others slightly vary your behavior before emulating it? I don't think so. You need negative feedback to change.

I have a background in physics and some electronics and signal analysis is part of that background. In electronics positive feedback doesn't have the credits that it has in social studies. It is viewed as a destructive force that will ruin your signal, make your appliance useless, or even destroy things. Let's look at the example of a feedback loop in it's most generic form:

Download now or preview on posterous

PastedGraphic-1.pdf (18 KB)

In the diagram you see input being processed by a system G the output of G is connected to its input via a feedback loop. If the feedback is negative or zero, the output will be strictly related to the input. If the input drops, the output goes to zero. If the feedback is positive two things can happen: first, if the feedback is less than the input, the output will eventually go to zero, but it takes more than one loop. In audio this is called an echo. It's not dangerous, but it can pollute the output if the echo is of a sound that you didn't want to hear. If the feedback signal is stronger than the input, when the input drops the output will continue to increase. In audio this is called a feedback loop, the typical symptom is a high frequency tone that gets louder and louder until either the speaker gives or the amplifier can't go any louder. This is not considered a good thing generally.

If we apply this model to human interaction, it still works. Let's say the input is an idea by developer Joe, G is the team. Let's say the idea is to improve quality by adding more test cases. Let's say the idea pays off the first loop and everybody is very happy. This causes some serious positive feedback. The feedback will say: "More testing is good". And more resources will be invested in testing. Depending on the positive attitude of the team this will lead to more and more resources being spent on creating tests, until there are no more tests to write or the team is out of resources. As much as I like tests, that's crazy. So developer Bill will come with the great idea to spend less time on testing. The first loop this will get positive feedback because the team gets stuff done again and the tests are in place to keep them safe. Two loops later we're in the hole again, quality is crap, coverage is abominable, we're stuck. Rinse and repeat. Does this really happen? You bet!

There is a dangerous psychological phenomenon that is entirely caused by positive feedback in groups, dubbed Group Think

The reason that people fall into this trap in my opinion is that they have the instincts to give positive feedback and act on it. Also our education system is based on it because it works well on small children. But in fact getting only positive feedback makes you lazy and stupid. What can we do? Simple: introduce negative feedback.

I thought negative feedback was bad?

If negative feedback means bashing then yes, it's bad. If negative feedback means using force, it's bad. If negative feedback is used to block others, it's bad. But it doesn't have to be that way. In the case were Joe suggested testing the first iteration good things happened, but then it spun out of control.

Download now or preview on posterous

PastedGraphic-5.pdf (18 KB)

The team lost track of the goal: working software and started focussing on some symptoms like test coverage.

In the next image the green arrow are the actions the team is taking, the red arrows are the negative feedback that is applied to reach their goal instead of overshooting it. 

Download now or preview on posterous

PastedGraphic-4.pdf (19 KB)

The important thing is not to stop the team from doing what it's doing, just to stop it from doing it too much. Make the team do it better instead of more. In some cases a really terrible idea comes up and no time should be wasted on it. In those cases people usually have no problem killing off the idea. The problem arises when a good idea comes up but the implementation is not completely right. It's only human to want to move forward when you're going strong and that makes it very easy to overshoot the goal entirely. The trick is to gradually apply more corrective negative feedback once the team gets closer to a goal.

Thinking about decision processes using this metaphor works for me. I hope you find it useful too.

Posted via email from iweinfuld's posterous

How to build Spring Integration from the sources

This is a guide on setting up your machine for Spring Integration development. I've got questions about this many times so I thought it was time I put the answer where Google could see it. Let's say you want to create a patch for a bug in Spring Integration, or you want to run a test case to see how some component works. First you need to take that annoying hurdle of getting the project to build locally. You're going in the right direction, but you've missed some turns. If that sounds familiar, read on!

Before you go into the IDE part, make sure you have got the basics right. I can't stress this enough, if you can't get it to work from the command line you're doomed, no tool can help you. So the 5 first steps don't involve an IDE.

Prepare your system for Spring Integration development
0. Forget about your IDE and open up a terminal (on Windows there are some extra steps).

1. make sure you have a command line svn client (there are many installation procedures) the end result should be:
(master) iwein:si$ svn --version
svn, version 1.6.11 (r934486)
   compiled Apr 19 2010, 23:04:06
Any version later than 1.6 will do I think.

2. make sure you have the right version of Java:
(master) iwein:si$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode)

This might be a bit confusing and I want to avoid the details, but you need to remember that Spring and also Spring Integration are compatible with Java 1.5, but the build is not necessarily. 1.6 is fine.

3. make sure you have the right version of Maven:
(master) iwein:elmar-trunk$ mvn -version
Maven version: 2.0.9
Java version: 1.6.0_15
OS name: "mac os x" version: "10.6" arch: "x86_64" Family: "mac"

Any 2.0.x version will do (except the ones with bugs :) ).

4. check out the trunk:
iwein:si-temp$ svn co https://src.springsource.org/svn/spring-integration/trunk/
This will give you a lot of output, but no errors.

5. Build the project:
iwein:si-temp$ cd trunk/
iwein:trunk$ mvn clean install
Again a lot of output, but no errors

6. Import the project in your IDE
Using the power of google you can find guides for:

• importing a maven project in IDEA
• using m2eclipse
• working with maven projects in NetBeans

It's a matter of taste how you do it, and if these fancy IDE's are not for you there is always vim. Point is that as long as you make sure everything works from the command line, the IDE support is just a convenience that you could live without (theoretically).

Other options
If you don't like Subversion, you can also use git. Make sure you have git and git-svn installed and then run:
git svn clone https://src.springsource.org/svn/spring-integration/trunk/ -r HEAD
If you don't know Subversion yet, I'd recommend this option.

If you are bored with Maven you could also check out the Spring Integration gradle build by Jeroen van Erp. Using a fork makes you dependent on Jeroen for porting commits on the Subversion HEAD, or your own merge skills, but the coolness of gradle might very well be worth it.

Let me know if you run into trouble or if you think I missed a step. Happy hacking!

Posted via email from iweinfuld's posterous

Simplifying Spring Integration testing

It seems like last week that I would happily write things like "public abstract class BaseMyCompanyTest extends AbstractTransactionalDatasourceSpringContextTests {" and think I was the man. It seems like yesterday that Spring 2.5 came along and made things a lot easier with @ContextConfiguration. But now I find myself grinding against the boilerplate again.

Before I start tearing it down let me first tell you that I highly respect the work by Sam Brannen and that I have gotten very good mileage out it for over two years now. There is always room for improvement and that is the aim of the following experiment.

The TestContext framework that comes with Spring might be very useful, it is also quite complex. You can only use it with a custom testrunner and and you need to pass the location of the configurations to be loaded in a class level annotation. This means it is non trivial and asymmetric to load multiple contexts in a single test. Also the incantations you have to put on the class are not exactly concise (although they are much better than the AbstractTooFreakinLongClassNames options of old).

Since JUnit 4.7 there is @Rule. And you can do some pretty cool stuff with it. When combining it with Spring though I wasn't impressed. I felt that Spring Test was making it hard to use the power of Rules. This is ironic, since it is just this JUnit feature that would make Spring Test a lot better.

How cool would it be if you could write:

1 2 3 4 5

@Rule   public TemporarySpringContext context = new TemporarySpringContext("context.xml");   @Autowired   ApplicationContext thisShouldBeWired;

And it would just work?

Gues what, I've got a green test that says it does! It only took me 50 lines of code, which you can find in my spring sandbox on github.

The important class is this the TemporarySpringContext (which I might give a better name soon):

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

public class TemporarySpringContext extends TestWatchman {   /** * Cache of Spring application contexts. This needs to be static, as tests * are typically destroyed and recreated between running individual test methods. */   static final ContextCache contextCache = new ContextCache();   private ConfigurableApplicationContext context;   private final String[] contextLocations;   public TemporarySpringContext(String... contextLocations) {     this.contextLocations = contextLocations;     try {       context = contextCache.contextForLocations(contextLocations);     } catch (Exception e) {       throw new RuntimeException(e);     }   }   @Override   public Statement apply(Statement base, FrameworkMethod method, Object target) {     context.getAutowireCapableBeanFactory().autowireBean(target);     return super.apply(base, method, target);   }   public ConfigurableApplicationContext getContext() {     return context;   }      public void dirtyContext(){     contextCache.markDirty(contextLocations);   } }

Because it extends TestWatchman you can hook into al the phases of your test by simply overriding a method. Because it is just a MethodRule field you can add different contexts to your test class. There is plenty of room to polish this, but it has more potential than the SpringJUnit4ClassRunner I think.

Since most of the heavy lifting is still being done by Spring Test behind the scenes by the way. The ContextCache is the only complex part (yet) but I have some tricks to pull to make @Transactional and such work. If you fix it before me I owe you a beer :)

Posted via email from iweinfuld's posterous

Excuse me, your fly is open

After I showed a colleague a trick in my IDE he told me to write a blog post about it. I told him to write it himself. This all happened almost a year ago. The problem with this post is that it's about something that is a bit embarrassing. I'm losing my appetite for talking about it, but yet, I come across more setups that are wrong than setups that are right. This post is mostly about some settings in Eclipse (and also in IntelliJ IDEA) that are pretty moronic and that you should change asap to prevent bugs in your code. Otherwise you might be seen with an open fly and you wouldn't want that would you?

First a word on keyboard shortcuts

First of all: learn to use your tool!  Here's a list of things that I see people wasting effort on:

- click through the menu (unless you're trying to find what the keyboard shortcut is you're looking for)

- declare a variable or field (use extract variable instead)

- type the skeleton of a method (your IDE can do that much faster using a quick fix)

- create a new type other than a test case yourself (generate everything from a test case, it's faster, trust me on this one)

There are many more variations on the same theme, the last point is arguable, some people are really fast if they start to hash out the model before beginning a test. My point is that it pays off to find the shortest path to a certain solution and figure out what shortcuts get you along that path fastest. The best way to train yourself to use the keyboard is to stick to the first point. There is even an eclipse plugin that can enforce keyboard usage (but that's a little too harsh for my taste). There are many blogs describing the keyboard shortcuts in great detail as well.

Templates

When you stop using your advanced typing skills and let the IDE do the typing for you, you will have moved on to using templates. In eclipse, if you look for templates under preferences (just type templates in the filter).

You will find quite a few templates that are useful, some that you'll never use and the best news is that you can create your own too. I can recommend renaming the JUnit4 template to tst (instead of the default Test in eclipse) and tweaking it a bit to your coding standards, but that's not why we're here now.

Not all these templates are sensible, and that is what I want to to fix.

New method body

A new method you can create by just trying to use the method and then asking Eclipse to create the missing method for you.

You'll end up with a method like this:

1 2 3 4 5 6 7 8

public class Reverser { public void reverse(String string) { // TODO Auto-generated method stub } }

Which is pretty bad because it will allow your code to run without complaining and leave a crappy piece of comment in there. It's not the end of the world though, and you'll notice later right? You probably won't, but you don't have to, more on that in a minute. First let's make the method return a String.

Stop, don't start modifying the method, just delete it. Eclipse will generate it correctly for you later.

Modify the test a bit so you need a String return value and try the quick fix again.

Now Eclipse gives you this:

1 2 3 4 5 6 7 8

public class Reverser { public String reverse(String string) { // TODO Auto-generated method stub return null; } }

This is not only pretty bad, its retarded. Just run the test case and see for yourself. The only plausible result of this will be a NPE, and the NPE will happen in another class so you have to retrace your steps to find out where you need to fix it. I want my test case to throw me a stack that starts exactly where I need to continue developing. Even worse, if you're generating methods that are not invoked directly from a test case, this might lead to NPE's flying around in production. This will not do.

So what can we do? Well, it's simple: just modify the template responsible. This template is called 'Method body' and should edit it to look something like this.

Now if you delete the generated method again and regenerate it, it should look better. 

1 2 3 4 5 6 7 8

public class Reverser { public String reverse(String string) { throw new UnsupportedOperationException( "Iwein left this implementation as an exercise to the reader"); } }

If we run the test case now it will fail. But the big improvement is that it will fail correctly. What I mean by failing correctly is that the failure actually points out what is wrong: you have failed to implement a generated method. This is infinitely more informative than an NPE. As a bonus you can click the stack trace in the test and actually go to the method that you need to implement.

Just double click the line that is selected in the above screenshot. Pretty cool huh?

Catch blocks

There is one more template I want you to modify. Check out the 'Catch block body' template. This is the template that gets used when you use the 'Surround with try/catch' quick fix. 

The resulting code looks like this:

1 2 3 4

} catch (Exception e) { // TODO Auto-generated catch block e.printStackTrace(); }

I can tell you that there is plenty of code like that running in production systems. Granted, checked exceptions are hard to deal with, but surely we can do better than this.

This isn't going to make your program run any better, but it sure is going to make it fail better. And that, dear reader is what this post is all about. If things fail early with an informative message, you (and those that come after you) are going to spend less time in a debugger and more time fixing actual problems. 

I've used Eclipse for the screenshots, but I can assure you that on my IDEA setup I have done the same fixes, and it wasn't very hard to figure out either. So now that you've read this there is no excuse to carelessly return null or print a stack trace anymore, even if you're not using Eclipse.

Don't be caught with an open fly again!

Posted via email from iweinfuld's posterous