Generally, there are four reasons why one might to set up a justice system and punish criminals for their crimes:
I) Retribution: basically the concept of punishment because one deserves it.
II) Deterrence: The threat of punishment prevents crime from being committed.
III) Protection of society: prevent the criminal from reoffending by locking him/her up or using capital punishment.
IV) Reform: Prevent the criminal from reoffending by reforming his/her behaviour and reintegrating them into normal society.
Risk and Reward
I think an important bearing on how to justify the justice system is the concept of risk and benefit. Your two main options to prevent reoffense at their most basic are locking people up for life, or reforming them and reintegrating them into society(I'm leaving capital punishment out of the debate atm). The main advantage of locking people up for life is that there is no chance for them to commit the same crime again(presuming they don't escape). The main disadvantages are that it is probably more expensive and some people might actually regret their actions and want to become law abiding members of society, but are not given the opportunity. The main advantages of reform are that it can rehabilitate the criminal and they can contribute to society, and it is probably less expensive. The main disadvantage is that it is possible to fake a reformation, get out, and commit the same crime over again.
In weighing up these risks one has got to consider the damage done by the crime itself. Although reform is good thing, there is a problem if its success rate is low and/or it is difficult to tell which criminals have reformed. However, with a crime like say shoplifting, it is acceptable to attempt reform and then release the criminals even if it is difficult to tell which criminals have reformed. This is because although shoplifting is a bad thing, the damage it does to people's lives is negligible. It is a tolerable risk that they will commit the crime again.
However, for violent crimes, particularly rape, murder, torture and child abuse, I would argue that the risk is not acceptable. If there is even a tiny chance that a criminal will commit the same crime again in this instance, he or she should be kept in jail for life. This is because these kinds of crimes are devastating to human life and human happiness. This is one reason I suggest life sentences without parole for these kinds of criminals.
Crime and Psychopathy
Success of reform is vital when taking it into account as a justification. If the success of reform is always 0% or its benefits are negligible it cannot be used as a justification.
This is partly why I believe that reform cannot always be considered important. For instance, many of the most violent criminals are psychopathic. Psychopathy is a condition in which one has an in-group containing only oneself and does not care about other human beings. One trait of the psychopath is that he/she cannot feel remorse, guilt or empathy. In my humble opinion there is no point trying to reform this kind of criminal. Why? Because reform has to start with convincing the criminal that what they did was wrong. If a criminal does not believe what they have done is wrong, they will just reoffend when let out of prison. Of course, psychopaths cannot or do not feel this remorse and guilt because they only care about themselves and as far as we know, there is no way to make them feel what they cannot.
Of course, psychopaths are often highly intelligent individual who could completely and convincingly pretend to reform without any moral remorse.
These arguments add to the weight of the ones above to suggest a harsh punishment for violent criminals.
(Admittedly not all individuals who commit a violent act are psychopathic, but in these most horrifying of cases, they mostly are. Even if they aren't, they have shown themself capable of committing these horrible acts when normal people would have been restrained by guilt or empathy.)
Deterrence and Human Nature
Human nature is not always a moral or wonderful thing. There are natural feelings we all(well mostly) have, moral or not. Unfortunately, elements of that human nature involve violence. If violence is a part of human nature, we need deterrents based on this.
It may seem counter intuitive to insist on harsh deterrents based on nature. We are often used to the idea of lesser punishments based on things that "aren't our fault" so to speak. (Many people try to exonerate others by appealing to the traumas etc in their childhood or whatever, although whatever you do you can't escape determinism.) But of course in this instance it should be the opposite: if something comes naturally to us, we need a threat of harsher punishment in order to abstain from doing it.
Reform and Deterrence as Joint Justifications
Even with crimes where there is a tolerable risk of reoffense, we need a deterrent to prevent the crime from happening in the first place, as well as reform system. As far as I can think of, the best system would be based partly on both these options for dealing with petty crimes. For the first offense, a limited sentence should be brought in, alongside a reform program. The criminals that are reformed are fine; but for the criminals that reoffend there should be a escalating system of deterrence, say the prison sentence for the second offense is double of that of the first offense, etc. This will make sure, not only that many criminals are reformed, but that even for criminals we cannot reform that they will eventually reach a deterrence level high enough for them to stop committing the crime.
For example, say for most want-to-be criminals a 3 month sentence for crime Y is a large enough deterrent, so that is put into practise. However, person X is an exception who only a sentence of 1 year or higher is enough to deter them. If we attempt to reform them after the first offense and it fails, next time they commit the same offense there will be a punishment of 6 months. The criminal X is now at a level where he/she is deterred, because a further crime would put them in prison for a year which is at a high enough level to be a deterrent.
Reform and Human Nature
Ideas on reform also need to take into account human nature. We cannot expect magical changes in somebody's fundamental psychological makeup, and to hope for it is wishful thinking and dangerous nonsense(see the psychopathy thing above, for instance.) However, facets of somebody's human nature may be able to be channelled into other activities more beneficial than crime.
Here is an example based on evolutionary psychology:
Men in general, complete for status. This is observed when we see people buying the latest phones and other gadgets to keep up with the Jones'. This is because, evolutionarily, a man at the top of the status heap would get more women and thus father more children than other men. A 17-year-old may compete for status by shoplifting so he can own the latest things. It may be possible to reform this criminal by channelling the evolutionary desire into a more productive hobby like sport.
Although this is oversimplified I think it demonstrates the ideas behind what would be good reform. It is more realistic than actually changing human nature and thus more likely to be effective.
Overall, the more violent and damaging the crime is, the more protection of society becomes important, and for the most violent crimes it outweighs the benefits of potential reform. Deterrence is required to help control our human nature, and good reform will work to channel it.
NB: I did not use retribution as a reason for a few reasons. Firstly, it is not a particularly moral justification and only satisfies our base desires, rather than being productive. It is also harder to justify than the other three on philosophical determinism.
NB2 - Capital Punishment: From a purely retributionist standpoint, I would say that capital punishment isn't harsh enough. capital punishment is however, the most effective way of protecting society(if they were locked up they could still technically escape.) Many people also think that it is a powerful deterrent but I personally disagree as I believe that death is the end of all sensation(as a materialist) and many criminals are religious(eg. Paul Hill wanted to be killed so he could get to heaven--no real deterrent there.) As a deterrent it is easier to argue that capital punishment isn't harsh enough, as torture would be more effective if unethical.