There are two components to this question: legal and ethical. I will address the legal issues first, and touch upon the ethical issues so that you guys can comment further.
Legally game rules are NOT protected. What is protected is copyright, trade name, trademark, and possibly design and innovation that can be patented. Rules do not fit into any of these.
Copyright protects the original creative work of an author or other artists. This includes written words, images, and music. Such work cannot be copied or attributed to someone else without permission. However, there are a lot of exceptions to this rule. Fair use is the biggest one. You are allowed to use “brief” excerpts in scholarly works if properly cited. You can parody a work to almost any degree. Music has become more complicated over the last few years, but thankfully game designers do not have to worry about those rules. At least for now. A musician can play any music or song without permission as long as it is a live performance and no recording is being sold. The same is the case for book readings.
A rough non-legal rule is that as long as you are not just wholly copying the work, and providing proper citation, it should be fine. If you take a set of rules and use the exact same mechanics, but you do not copy the text, images, trade name, or trademark then there is no violation, you do not need any permission, and you do not have to provide any attribution.
Trade name and Trademark should be fairly easy to understand. Do not use other people’s names or logos without their permission. But what if the names and logos have become part of the Public Domain like chess or checkers? Then it is open season.
Patent is another issue:
A patent is a set of exclusive rights granted by a sovereign state or intergovernmental organization to an inventor or assignee for a limited period of time in exchange for detailed public disclosure of an invention. An invention is a solution to a specific technological problem and is a product or a process.
Wikipedia
I am not really sure how a set of rules or a rule mechanic could become a patentable technology or process, but the courts have been recognizing “process” patents that would never have been granted before: Priceline was granted patent protection for its use of the age old Dutch-auction system in its online system.
In conclusion, as of today, game rules are not protected. Game text, images, trade names, and trademarks are protected unless they have become part of the Public Domain.
Ethical Considerations
No one wants to be a jerk. At least most of us do not want to be a jerk. Also, it might be bad for your sales if people think you are just “stealing” the work of a beloved and respected founder of the gaming community.
However, for me there is a difference to “hacking” a game and “stealing” it. One of my favorite games is De Bellis Antiquitatus. This game was designed in the 1990s as one of the first “fast play” wargames. It provided a very short yet compelling, if somewhat abstract, set of rules for ancient battles. I took the rules and changed them for the age of muskets. When I was finished I had created a very different game built upon the foundation of DBA. This practice is so common that I think it is silly to discourage it.
On the other hand, what if a set of rules goes out of print, and you want to recreate it word for word with full attribution to the author? That would be a violation of copyright law unless you obtained prior permission, and most authors will not agree to this since they are hoping someone will pay them to put it back into print, or they are working on a revised edition and they do not want the old version competing with their new version. So you cannot just reprint something even if it is out of print. Very irritating if you ask me. Ironically, you CAN “hack” a similar set of rules that work just the same, uses the same game mechanics, but is not an outright copy of the rules. Many people have done this with D&D.
I had an idea on how to not only hack ‘Hordes of the Things, 2nd Edition’ (which is essentially out of print right now), but also add extra content in order to create a more comprehensive game system with lots of neophyte friendly pictures, diagrams, and explanations on how to not only play the game but to also create and paint your armies. I was going to name it ‘Multitudes of Monsters and Mages’, but the HOTT community reacted so negatively to this that I decided to stop. Oddly enough the author was more than happy to see me do this, and offered to help but would not officially approve the project. Now the game is being distributed by “illegal” photocopies and PDFs.