「of course, no, you can't be a formal member.」 - This isn't always the case.
The rules depend on the individual club. Some clubs only accept members from the school where the club is based, and others accept students from other clubs as well. When I was studying at a university in Tokyo, some of the clubs even had policies where only men from the school could join, but women from any school could join. I was personally part of a club that had members from four or five different schools which all had campuses fairly close to each other.
Some of the previous posters have mentioned the terms "circle" (サークル) and "dokokai" (同好会, literally "group that likes the same thing"), which, along with "club" (クラブ) and "bu" (部) are all words that can be used to mean "school club" or "school team" in Japan. In many Japanese universities, there will be multiple clubs for the same activity, often with different levels of how seriously they take the activity and how often they meet/practice.
For sports, generally the "bu" is what's considered the formal "team," and so those are almost always open only to students attending that university. For example, the Waseda University baseball-bu only accepts players who are attending Waseda University, trains intensely, and represents the school in official competitions/tournaments against other schools.
However, there are a lot of students who like playing baseball, but either can't or don't want to spend the amount of time practicing that the baseball-bu requires, or maybe simply aren't talented enough to make the team. But since they still want to play baseball, they might form a more relaxed baseball club. Maybe they don't practice every week. Maybe they don't have uniforms. But they're a group that gets together and play baseball from time to time, and forming a club is easier than just randomly trying to call up 17 other people you happen to know and see if they want to play a game. It's generally these lower-tier clubs (i.e. not the -bu) that sometimes accept members from outside the school.
At the school I went to, for example, there were around 20 different tennis clubs, with the tennis-bu at the top with people who had a shot at going pro (and the practice schedules to match), all the way down to a group that got together for an hour a month to hit balls on a court with no referees and then went out to eat and drink afterwards.
Some universities have directories that list all their clubs, and these will likely the membership eligibility for each club, but if you can't find such information, there's no harm in simply asking the club directly if they accept outside members. Bear in mind, though, that most university clubs do at least require members to be students. So, for example, if you're working, not studying, in Japan, odds are you won't be able to join.