"Old man balls" are a real thing, and there's not much you can do about it. Time and gravity will always win in the end.
It happens to tall of us as we get older, though at different rates and times and extents. But inevitably, whether you started your adulthood with a high and tight ballsack or you already were dealing with low swinging balls, they will hang lower and lower as the years go by. For most men it's a gradual process, just like graying hair, wrinkles, and more — all natural parts of aging. In fact, the sagging of your balls is caused by the same factors that cause the wrinkling and sagging of skin around your body: time, gravity, and decreasing elasticity.
It helps to understand scrotal anatomy. At the most basic level, the scrotum is a sack of smooth cremaster muscle, with stretchy skin on the outside and your testicles and associated plumbing on the inside. Thanks a quirk of biology, testicles work best at 96° F (35.6° C), so they sit outside our warmer torso to stay cool. Because the temperature outside the body is a lot more variable, that scrotal muscle sack will contract to bring the testicles closer to the body when it's cold, and relax to move them away when it's warm. This is why you typically hang lower during the summer versus the winter.
As with all muscles on your body, the older you get the weaker the cremaster muscle becomes. There are some unsupported claims that you can exercise these muscles by doing kegels (flexing the muscle that you use to cut off urine flow, which will strengthen your pelvic floor), but the truth is that time and gravity will always win. In addition to weakened scrotal muscles, skin all over your body gradually loses elasticity as you age. So while your balls may still retract all the way up when it's cold, the scrotal skin won't provide as resilient of a stop when it's warm and they sag.
Benign medical conditions can also exacerbated a saggy ballsack. A build-up of fluid in the scrotum called a hydrocele can literally weigh down the scrotum and cause stretching, though it's generally harmless and painless. The valves in the veins that drain blood away from the testicles can malfunction and cause a backup of blood called varicocele, which will increase temperature in the scrotum and cause the muscles to relax in a vain effort to keep things cool. Both can be treated if they are extreme cases, but many men will live with them without inconvenience or treatment.
More urgent medical conditions can cause a more sudden scrotal hanging. A hernia can lead to a bit of your intestines dropping into the scrotum, increasing both the mass inside the sack and the heat, causing it to sag lower. Testicular cancer can also increase heat by demanding additional blood flow. Injuries to the testicles and scrotum can also disrupt blood flow, causing the body to overcompensate and pump more than is needed, increasing heat and making it all hanging more to stay cool.
While scrotal sag itself isn't an issue, it can still cause discomfort for some men. For one, having balls hanging down where they weren't before means they can get in the way of everyday activities and cause bulging in new ways. The first time that I sat on my own lower-hanging balls is a moment that is forever etched in my memory. Low-hanging balls can also cause physical pain, as the weakened muscle and skin are stretched by their weight and gravity, leading to aching in the scrotum and up into the pelvis. There can even be psychological effects from lower-hanging nuts, as they're yet another sign that you're not as young and resilient as you once were and for some men that can be very hurtful to their self image.
Unfortunately, there's not much you can do to stop scrotal sag as you age. You might think that wearing well-fitting and supportive underwear would abate the issue, but the main factors of weakened scrotal muscle and reduced skin elasticity will happen regardless of whether you're holding your balls up or not. At least supportive underwear can alleviate hanging-related scrotal aches by stopping your nuts from actively pulling on the scrotum.