Model Boat Mayhem

Please login or register.

Login with username, password and session length.
Pages: 1 [2] 3 4   Go Down

Author Topic: Magazine article Database?  (Read 14072 times)

Peterm

  • Inactive
  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 414
  • Location: Southam,Warwickshire
Re: Magazine article Database?
« Reply #25 on: February 06, 2009, 03:00:29 pm »

Martin, you are brilliant!  Pete M
Logged
I`m not just old, I`m ancient

Colin Bishop

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 12,188
  • Location: SW Surrey, UK
Re: Magazine article Database?
« Reply #26 on: February 06, 2009, 03:04:10 pm »

Different people will enter information in different ways so when you search you will only get a proportion of what is actually there and quite possibly miss the bits you were actually looking for.  You will only get a patchwork of articles and is there anything to stop the same one being entered more than once? No data validation for consistent spelling. Sorry Martin, but this is emphatically not the way I was taught to put a database together - but good luck anyway!  {:-{

Colin
Logged

Martin (Admin)

  • Administrator
  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 23,422
  • Location: Peterborough, UK
    • Model Boat Mayhem
Re: Magazine article Database?
« Reply #27 on: February 06, 2009, 03:29:56 pm »

It's getting the data that the hard work.
Once we got the data starts coming in, we can sort it all out as we go along....  :-))
Logged
"This is my firm opinion, but what do I know?!" -  Visit the Mayhem FaceBook Groups!  &  Giant Models

Colin Bishop

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 12,188
  • Location: SW Surrey, UK
Re: Magazine article Database?
« Reply #28 on: February 06, 2009, 03:31:14 pm »

Quote
Once we got the data we can sort it all out as we go along.... 

Oh Martin! Famous, Famous last words...... {-)
Logged

Martin (Admin)

  • Administrator
  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 23,422
  • Location: Peterborough, UK
    • Model Boat Mayhem
Re: Magazine article Database?
« Reply #29 on: February 06, 2009, 03:33:22 pm »

I'm Famous for my Famous last words!  :D
Logged
"This is my firm opinion, but what do I know?!" -  Visit the Mayhem FaceBook Groups!  &  Giant Models

tigertiger

  • Global Moderator
  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 7,748
  • Location: Kunming, city of eternal springtime, SW China.
Re: Magazine article Database?
« Reply #30 on: February 06, 2009, 03:36:13 pm »

The database design and the data definition are the important bits.

Otherwise you cannot mine the data effectively after it has been captured.

Quick and dirty solutions are not scaleable.
And this thing will grow and so I think scalability is a major issue.
Logged
The only stupid question is the one I didn't ask

tigertiger

  • Global Moderator
  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 7,748
  • Location: Kunming, city of eternal springtime, SW China.
Re: Magazine article Database?
« Reply #31 on: February 06, 2009, 03:37:13 pm »

I'm Famous for my Famous last words!  :D

Was that
'Kiss me Hardy'  :kiss: or
'Et tu Brute'  {-)
Logged
The only stupid question is the one I didn't ask

Colin Bishop

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 12,188
  • Location: SW Surrey, UK
Re: Magazine article Database?
« Reply #32 on: February 06, 2009, 03:42:42 pm »

No it was "They told me it would be OK...."

or maybe "Yes sir! Your money is perfectly safe with us." {-)
Logged

DickyD

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 9,423
  • www.srcmbc.org.uk
  • Location: Southampton UK
    • SRCMBC
Re: Magazine article Database?
« Reply #33 on: February 06, 2009, 03:49:46 pm »

No, it was " Ctrl+F5 will solve all your problems  >:-o
Logged
Richard Solent Radio Controlled Model Boat Club http://www.srcmbc.org.uk

Martin (Admin)

  • Administrator
  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 23,422
  • Location: Peterborough, UK
    • Model Boat Mayhem
Re: Magazine article Database?
« Reply #34 on: February 06, 2009, 03:54:32 pm »

The database design and the data definition are the important bits.

Otherwise you cannot mine the data effectively after it has been captured.

Quick and dirty solutions are not scaleable.
And this thing will grow and so I think scalability is a major issue.

Agreed. Anybody want to step up?

All I think most people want, well me for one is a searchable database where I can type in eg. sandpiper or planking and
I get some results back. At the moment we have nothing and starting with a spreadsheet is easily expandable and
convertible to a database with a minimum of effort. I can add, remove or change any field as quick as you like but
 - lets get the data in first!<*<
Logged
"This is my firm opinion, but what do I know?!" -  Visit the Mayhem FaceBook Groups!  &  Giant Models

Bryan Young

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 6,883
  • Location: Whitley Bay
Re: Magazine article Database?
« Reply #35 on: February 06, 2009, 05:03:37 pm »

Oh my, goodness gracious! What have I started now?
I certainly didn't expect this level of response....but it seems to have triggered a need (desire?) for some sort of access to info that isn't readily available to all.
May I give you a little background to all this? For my sins I have been looking after the records etc. of Tynemouth Model Boat Club for a number of years now. Including "Model Boats", "Marine Modelling" and sundry others. About 7 ft of shelf space. Now I need the space back.The "take-up" of requests from club members has been abysmal, so I decided to go down the "putting it all on disc" route. I now have 12 CDs (1967-2000), and the annotated contents pages on another disc. All free to club members. But perusing all this still requires some reading....and I'm sure you all agree that the "average" club member of anything wants things presented on a plate. Hence the request for help with a database. I also hold a shed load of plans (not all TMBC property) that I would like to be able to disseminate.
I have a suggestion.  The committe of TMBC seem to have gone into the shaking of the head, tooth sucking mode, wondering what to do with this pile of stuff. It isn't really saleable as it is by no means 100% complete. If Martin or anyone who will be visiting the NE Model Boat show who is passing(ish) Martins castle is willing then the mags can be passed over to be done properly....rather than finish up in a skip. I will try to convince our committee that this would be "a good thing".Ideas?
But thank you all for the staggering response to what I thought was a simple question.
For Dicky and Stavros.....yes, I have included the kit revues and builds. But lets not go there again! Bryan.
Logged
Notes from a simple seaman

Colin Bishop

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 12,188
  • Location: SW Surrey, UK
Re: Magazine article Database?
« Reply #36 on: February 06, 2009, 05:22:30 pm »

Well, there's your data Martin....  {-)
Logged

Bryan Young

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 6,883
  • Location: Whitley Bay
Re: Magazine article Database?
« Reply #37 on: February 06, 2009, 05:31:33 pm »

Oh my, goodness gracious! What have I started now?
I certainly didn't expect this level of response....but it seems to have triggered a need (desire?) for some sort of access to info that isn't readily available to all.
May I give you a little background to all this? For my sins I have been looking after the records etc. of Tynemouth Model Boat Club for a number of years now. Including "Model Boats", "Marine Modelling" and sundry others. About 7 ft of shelf space. Now I need the space back.The "take-up" of requests from club members has been abysmal, so I decided to go down the "putting it all on disc" route. I now have 12 CDs (1967-2000), and the annotated contents pages on another disc. All free to club members. But perusing all this still requires some reading....and I'm sure you all agree that the "average" club member of anything wants things presented on a plate. Hence the request for help with a database. I also hold a shed load of plans (not all TMBC property) that I would like to be able to disseminate.
I have a suggestion.  The committe of TMBC seem to have gone into the shaking of the head, tooth sucking mode, wondering what to do with this pile of stuff. It isn't really saleable as it is by no means 100% complete. If Martin or anyone who will be visiting the NE Model Boat show who is passing(ish) Martins castle is willing then the mags can be passed over to be done properly....rather than finish up in a skip. I will try to convince our committee that this would be "a good thing".Ideas?
But thank you all for the staggering response to what I thought was a simple question.
For Dicky and Stavros.....yes, I have included the kit revues and builds. But lets not go there again! Bryan.
[/quote
I suppose the word "revue" conjures up visions of lap dancing.....perhaps my mind wandered....I shal review the situation.
Logged
Notes from a simple seaman

w3bby

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 773
  • Location: Helsingborg, Sweden
Re: Magazine article Database?
« Reply #38 on: February 06, 2009, 06:56:05 pm »

I shal review the situation.
Mind wandering again Bryan  :}

chingdevil

  • Guest
Re: Magazine article Database?
« Reply #39 on: February 06, 2009, 07:06:12 pm »

Martin

Who is going to control the database and check for consistency of entries, perhaps control is the wrong word. Whe is going to be the person to contact if there is a problem with the database? Also who is going to be the person who has the access to correct any problems with the database?


Brian
Logged

Bee

  • Guest
Re: Magazine article Database?
« Reply #40 on: February 06, 2009, 07:13:07 pm »

for some threads everyone knows a little bit  :}
Avoid database programs like the plague.

I think Martin's first post was good, except each field should be separated by a comma and ensure no commas elsewhere. This can produce a CSV (comma separated values) file that excel and many other programs will input. (do they have CR/LF at the end of a section?)

Martin's google stuff offers one way of capturing data, others may prefer Notepad, others Access. Doesn't matter provided they can all output a CSV.
The CSVs can be merged.
Spelling errors don't matter becasue spell checkers can clear it up ( a feature not often available in a fancy database).
If the data is simple and standard it can then be post processed and presented in many different ways.
Data may not be perfect and articles missed but it's not a safety critical system - so don't fuss.
Programmers can offer different programs for searching over time to improve capturing similar words and spelling or import it into the database of their choice.
Dates are probably the thing that will end up with most different formats, but they can be processed into a standard.

Data entry - snow bound, ill, disabled, bored, retired, lots of us have an hour to fill. So what if there are dupications if it tells you twice it won't hurt and eventually the data can be cleaned.
Logged

Bee

  • Guest
Re: Magazine article Database?
« Reply #41 on: February 06, 2009, 07:23:55 pm »

yes where to store.
Can a non-picture file be attached to a post? If too big it could be divided into decades perhaps. There could be a thread per decade. After a while once a newer file was 'verified' by useres the oldest version could be expunged to keep the size under control. Somone posting a new version would just say what they had changed in the post - self documenting.
Say 10 years = 120 issues = 2400 items(at 20 items per issue) = 240k at 100 characters per item. Not too big. Mostly user administered.
Logged

Colin Bishop

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 12,188
  • Location: SW Surrey, UK
Re: Magazine article Database?
« Reply #42 on: February 06, 2009, 07:26:38 pm »

Quote
eventually the data can be cleaned

And I've spent many unhappy hours in the past just doing that to the extent that I'm convinced it's better to put in the upfront effort to get it right in the first place.

Colin
Logged

Bryan Young

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 6,883
  • Location: Whitley Bay
Re: Magazine article Database?
« Reply #43 on: February 06, 2009, 07:43:47 pm »

Quote
eventually the data can be cleaned

And I've spent many unhappy hours in the past just doing that to the extent that I'm convinced it's better to put in the upfront effort to get it right in the first place.

Colin
Whats that mean... in English.
Logged
Notes from a simple seaman

Colin Bishop

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 12,188
  • Location: SW Surrey, UK
Re: Magazine article Database?
« Reply #44 on: February 06, 2009, 07:59:28 pm »

If a job is worth doing it's worth doing properly Bryan.
Logged

Bee

  • Guest
Re: Magazine article Database?
« Reply #45 on: February 06, 2009, 08:28:21 pm »

Quote
eventually the data can be cleaned

And I've spent many unhappy hours in the past just doing that to the extent that I'm convinced it's better to put in the upfront effort to get it right in the first place.

Colin
Whats that mean... in English.
When the data is entered someone might keep putting in "Brian Yung" which will mean simple searches fail for Bryan. For a professional application like Criminal Records this would be unacceptable. Either you say nobody can enter data enless they are a member of the 'faultless data entry union' or you have someone clean up the errors afterwards. For our application the first would yield zero volunteers so forget it.
It's not nice to ignore Bryan's articles but not the end of the world. Sometime when it is realised someone can do a simple search and replace. The great thing is that if the data is in a simple form anyone can load it into Notepad and do that then post the corrected version with a note to say what they did. It won't even need a script programmer.
The only big one we should get right in the first place is the date since 6/2/09 and 2/6/09 are both valid but one is wrong and it is only possible to correct by looking at the article in question. I suggest we insist on 3 letter months since 6/Feb/09 and Feb/6/09 are unambiguous and anyway Martin's entry form can force it to be in this form. If some entries don't use the form and end up the wrong way round they will need correcting to make it easier for searches. But again there must be a dozen programmers on the forum who can do that.
Logged

Colin Bishop

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 12,188
  • Location: SW Surrey, UK
Re: Magazine article Database?
« Reply #46 on: February 06, 2009, 08:41:40 pm »

Yes, but in the real world people type in Brain Yung, Byryan Yurng and several other permutations besides, And they do it for all the other entries too so you end up with a huge number of inconsistent entries which take forever to rationalise using search and replace (if you can guess some of the more arcane versions anyway). I have been in this situation with a database of outstanding building maintenance at hundreds of properties involving thousands of entries typed in by individual surveyors. These had to be sorted into various categories to meet Government statistical requirements identifying the outstanding maintenance backlog between local authorities. We had to do it to get the Government grants but it was a dreadful job and the number of ways the surveyors spelt or described the same things was simply incredible. With a bit of data entry validation programmed into the input routines most of this could have been avoided. I am consequently a great believer in preventing a mess from developing in the first place rather than in clearing it up afterwards but I appear to be in the minority here! Been there, done that, not getting caught again!

Colin
Logged

Martin (Admin)

  • Administrator
  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 23,422
  • Location: Peterborough, UK
    • Model Boat Mayhem
Re: Magazine article Database?
« Reply #47 on: February 06, 2009, 08:45:46 pm »

Bryan, what "fields" (columns?) does you data consist of?

After years of nagging at people I can't believe I'm actually saying this but in this case I'm actually in favour of using
a spreadsheet data collection over a database. Data, sorting, duplicates, spell checking, errors, etc can be quickly
spotted and corrected in a spreadsheet and to some extent, done automatically.
Logged
"This is my firm opinion, but what do I know?!" -  Visit the Mayhem FaceBook Groups!  &  Giant Models

Colin Bishop

  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 12,188
  • Location: SW Surrey, UK
Re: Magazine article Database?
« Reply #48 on: February 06, 2009, 08:49:10 pm »

Looking at Bryan's post Martin I think he has simply scanned in the articles. but I could be wrong.  :o

Colin
Logged

Martin (Admin)

  • Administrator
  • Full Mayhemer
  • *****
  • Offline Offline
  • Posts: 23,422
  • Location: Peterborough, UK
    • Model Boat Mayhem
Re: Magazine article Database?
« Reply #49 on: February 06, 2009, 08:54:05 pm »

Quote
   a great believer in preventing a mess from developing in the first place   

Hmmm, good idea Colin, I'm not one known for that kind of rational thinking!  %)
How about we get some records together, say 50 or 100, we can then assess what we have and see if it's worth pursuing!

Go on Colin, enter one article.... even if it's just to prove me wrong!  O0
Logged
"This is my firm opinion, but what do I know?!" -  Visit the Mayhem FaceBook Groups!  &  Giant Models
Pages: 1 [2] 3 4   Go Up
 

Page created in 0.112 seconds with 22 queries.